--- name: alibabacloud-dataworks-datastudio-develop description: | DataWorks data development Skill. Create, configure, validate, deploy, update, move, and rename nodes and workflows. Manage components, file resources, and UDF functions. Covers 150+ node types: Shell, SQL, Python, DI, Flink, EMR, etc. Supports scheduled and manual workflow orchestration via aliyun CLI or Python SDK. WARNING: Supports mutating operations (Move, Rename) requiring explicit user confirmation. Delete operations are NOT supported by this skill. Triggers: DataWorks, data development nodes, workflows, FlowSpec, scheduling tasks, data integration, ETL pipelines, .spec.json. Also triggers for Alibaba Cloud data development, scheduling node configuration, FlowSpec format, or DI task orchestration. --- # DataWorks Data Development ## ⚡ MANDATORY: Read Before Any API Call **These absolute rules are NOT optional — violating ANY ONE means the task WILL FAIL:** 0. **FIRST THING: Switch CLI profile.** Before ANY `aliyun` command, run `aliyun configure list`. If multiple profiles exist, run `aliyun configure switch --profile ` to select the correct one. Priority: prefer a profile whose name contains `dataworks` (case-insensitive); otherwise use `default`. **Do NOT skip this step. Do NOT run any `aliyun dataworks-public` command before switching.** NEVER read/echo/print AK/SK values. 1. **NEVER install plugins.** If `aliyun help` shows "Plugin available but not installed" for dataworks-public → **IGNORE IT**. Do NOT run `aliyun plugin install`. PascalCase RPC works without plugins (requires CLI >= 3.3.1). 2. **ONLY use PascalCase RPC.** Every DataWorks API call must look like: `aliyun dataworks-public CreateNode --ProjectId ... --Spec '...'`. Never use kebab-case (`create-file`, `create-node`, `create-business`). 3. **ONLY use these APIs for create:** `CreateWorkflowDefinition` → `CreateNode` (per node, with `--ContainerId`) → `CreatePipelineRun` (to deploy). 4. **ONLY use these APIs for update:** `UpdateNode` (incremental, `kind:Node`) → `CreatePipelineRun` (to deploy). Never use `ImportWorkflowDefinition`, `DeployFile`, or `SubmitFile` for updates or publishing. 4a. **ONLY use these APIs for deploy/publish:** `CreatePipelineRun` (Type=Online, ObjectIds=[ID]) → `GetPipelineRun` (poll) → `ExecPipelineRunStage` (advance). **NEVER use** `DeployFile`, `SubmitFile`, `ListDeploymentPackages`, or `GetDeploymentPackage` — these are all legacy APIs that will fail. 5. **If `CreateWorkflowDefinition` or `CreateNode` returns an error, FIX THE SPEC — do NOT fall back to legacy APIs.** Error 58014884415 means your FlowSpec JSON format is wrong (e.g., used `"kind":"Workflow"` instead of `"kind":"CycleWorkflow"`, or `"apiVersion"` instead of `"version"`). Copy the exact Spec from the Quick Start below. 6. **Run CLI commands directly — do NOT create wrapper scripts.** Never create `.sh` scripts to batch API calls. Run each `aliyun` command directly in the shell. Wrapper scripts add complexity and obscure errors. 7. **Saving files locally is NOT completion.** The task is only done when the API returns a success response (e.g., `{"Id": "..."}` from `CreateWorkflowDefinition`/`CreateNode`). Writing JSON files to disk without calling the API means the workflow/node was NOT created. Never claim success without a real API response. 8. **NEVER simulate, mock, or fabricate API responses.** If credentials are missing, the CLI is misconfigured, or an API call returns an error — report the exact error message to the user and **STOP**. Do NOT generate fake JSON responses, write simulation documents, echo hardcoded output, or claim success in any form. A simulated success is worse than an explicit failure. 9. **Credential failure = hard stop.** If `aliyun configure list` shows empty or invalid credentials, or any CLI call returns `InvalidAccessKeyId`, `access_key_id must be assigned`, or similar auth errors — **STOP immediately**. Tell the user to configure valid credentials outside this session. Do NOT attempt workarounds (writing config.json manually, using placeholder credentials, proceeding without auth). No subsequent API calls may be attempted until credentials are verified working. 10. **ONLY use APIs listed in this document.** Every API you call must appear in the API Quick Reference table below. If you need an operation that is not listed, check the table again — the operation likely exists under a different name. **NEVER invent API names** (e.g., `CreateDeployment`, `ApproveDeployment`, `DeployNode` do NOT exist). If you cannot find the right API, ask the user. **If you catch yourself typing ANY of these, STOP IMMEDIATELY and re-read the Quick Start below:** `create-file`, `create-business`, `create-folder`, `CreateFolder`, `CreateFile`, `UpdateFile`, `plugin install`, `--file-type`, `/bizroot`, `/workflowroot`, `DeployFile`, `SubmitFile`, `ListFiles`, `GetFile`, `ListDeploymentPackages`, `GetDeploymentPackage`, `CreateDeployment`, `ApproveDeployment`, `DeployNode`, `CreateFlow`, `CreateFileDepends`, `CreateSchedule` ## ⛔ Prohibited Legacy APIs This skill uses DataWorks OpenAPI version **2024-05-18**. The following legacy APIs and patterns are **strictly prohibited**: | Prohibited Legacy Operation | Correct Replacement | |----------------|----------| | `create-file` / `CreateFile` (with `--file-type` numeric type code) | `CreateNode` + FlowSpec JSON | | `create-folder` / `CreateFolder` | No folder needed, use `CreateNode` directly | | `create-business` / `CreateBusiness` / `CreateFlowProject` | `CreateWorkflowDefinition` + FlowSpec | | `list-folders` / `ListFolders` | `ListNodes` / `ListWorkflowDefinitions` | | `import-workflow-definition` / `ImportWorkflowDefinition` (for create or update) | `CreateWorkflowDefinition` + individual `CreateNode` calls (for create); `UpdateNode` per node (for update) | | Any operation based on folder paths (`/bizroot`, `/workflowroot`, `/Business Flow`) | Specify path via `script.path` in FlowSpec | | `SubmitFile` / `DeployFile` / `GetDeploymentPackage` / `ListDeploymentPackages` | `CreatePipelineRun` + `ExecPipelineRunStage` | | `UpdateFile` (legacy file update) | `UpdateNode` + FlowSpec JSON (`kind:Node`, incremental) | | `ListFiles` / `GetFile` (legacy file model) | `ListNodes` / `GetNode` | | `aliyun plugin install --names dataworks-public` (legacy plugin) | No plugin installation needed, use PascalCase RPC direct invocation | **How to tell — STOP if any of these are true**: - You are typing `create-file`, `create-business`, `create-folder`, or any kebab-case DataWorks command → **WRONG**. Use PascalCase RPC: `CreateNode`, `CreateWorkflowDefinition` - You are running `aliyun plugin install` → **WRONG**. No plugin needed; PascalCase RPC direct invocation works out of the box (requires CLI >= 3.3.1) - You are constructing folder paths (`/bizroot`, `/workflowroot`) → **WRONG**. Use `script.path` in FlowSpec - Your FlowSpec contains `apiVersion`, `type` (at node level), or `schedule` → **WRONG**. See the correct format below > **CLI Format**: ALL DataWorks 2024-05-18 API calls use **PascalCase RPC direct invocation**: > `aliyun dataworks-public CreateNode --ProjectId ... --Spec '...' --user-agent AlibabaCloud-Agent-Skills` > This requires `aliyun` CLI >= 3.3.1. No plugin installation is needed. ### ⚠️ FlowSpec Anti-Patterns Agents commonly invent wrong FlowSpec fields. The correct format is shown in the Quick Start below. | ❌ WRONG | ✅ CORRECT | Notes | |----------|-----------|-------| | `"apiVersion": "v1"` or `"apiVersion": "dataworks.aliyun.com/v1"` | `"version": "2.0.0"` | FlowSpec uses `version`, not `apiVersion` | | `"kind": "Flow"` or `"kind": "Workflow"` | `"kind": "CycleWorkflow"` (for workflows) or `"kind": "Node"` (for nodes) | Only `Node`, `CycleWorkflow`, `ManualWorkflow` are valid. `"Workflow"` alone is NOT valid | | `"metadata": {"name": "..."}` | `"spec": {"workflows": [{"name": "..."}]}` | FlowSpec has no `metadata` field; name goes inside `spec.workflows[0]` or `spec.nodes[0]` | | `"type": "SHELL"` (at node level) | `"script": {"runtime": {"command": "DIDE_SHELL"}}` | Node type goes in `script.runtime.command` | | `"schedule": {"cron": "..."}` | `"trigger": {"cron": "...", "type": "Scheduler"}` | Scheduling uses `trigger`, not `schedule` | | `"script": {"content": "..."}` without `path` | `"script": {"path": "node_name", ...}` | `script.path` is always required | ### 🚀 Quick Start: End-to-End Workflow Creation Complete working example — create a scheduled workflow with 2 dependent nodes: ```bash # Step 1: Create the workflow container aliyun dataworks-public CreateWorkflowDefinition \ --ProjectId 585549 \ --Spec '{"version":"2.0.0","kind":"CycleWorkflow","spec":{"workflows":[{"name":"my_etl_workflow","script":{"path":"my_etl_workflow","runtime":{"command":"WORKFLOW"}}}]}}' \ --user-agent AlibabaCloud-Agent-Skills # → Returns {"Id": "WORKFLOW_ID", ...} # Step 2: Create upstream node (Shell) inside the workflow # IMPORTANT: Before creating, verify output name "my_project.check_data" is not already used by another node (ListNodes) aliyun dataworks-public CreateNode \ --ProjectId 585549 \ --Scene DATAWORKS_PROJECT \ --ContainerId WORKFLOW_ID \ --Spec '{"version":"2.0.0","kind":"Node","spec":{"nodes":[{"name":"check_data","id":"check_data","script":{"path":"check_data","runtime":{"command":"DIDE_SHELL"},"content":"#!/bin/bash\necho done"},"outputs":{"nodeOutputs":[{"data":"my_project.check_data","artifactType":"NodeOutput"}]}}]}}' \ --user-agent AlibabaCloud-Agent-Skills # → Returns {"Id": "NODE_A_ID", ...} # Step 3: Create downstream node (SQL) with dependency on upstream # NOTE on dependencies: "nodeId" is the CURRENT node's name (self-reference), "output" is the UPSTREAM node's output aliyun dataworks-public CreateNode \ --ProjectId 585549 \ --Scene DATAWORKS_PROJECT \ --ContainerId WORKFLOW_ID \ --Spec '{"version":"2.0.0","kind":"Node","spec":{"nodes":[{"name":"transform_data","id":"transform_data","script":{"path":"transform_data","runtime":{"command":"ODPS_SQL"},"content":"SELECT 1;"},"outputs":{"nodeOutputs":[{"data":"my_project.transform_data","artifactType":"NodeOutput"}]}}],"dependencies":[{"nodeId":"transform_data","depends":[{"type":"Normal","output":"my_project.check_data"}]}]}}' \ --user-agent AlibabaCloud-Agent-Skills # Step 4: Set workflow schedule (daily at 00:30) aliyun dataworks-public UpdateWorkflowDefinition \ --ProjectId 585549 \ --Id WORKFLOW_ID \ --Spec '{"version":"2.0.0","kind":"CycleWorkflow","spec":{"workflows":[{"name":"my_etl_workflow","script":{"path":"my_etl_workflow","runtime":{"command":"WORKFLOW"}},"trigger":{"cron":"00 30 00 * * ?","timezone":"Asia/Shanghai","type":"Scheduler"}}]}}' \ --user-agent AlibabaCloud-Agent-Skills # Step 5: Deploy the workflow online (REQUIRED — workflow is not active until deployed) aliyun dataworks-public CreatePipelineRun \ --ProjectId 585549 \ --Type Online --ObjectIds '["WORKFLOW_ID"]' \ --user-agent AlibabaCloud-Agent-Skills # → Returns {"Id": "PIPELINE_RUN_ID", ...} # Then poll GetPipelineRun and advance stages with ExecPipelineRunStage # (see "Publishing and Deploying" section below for full polling flow) ``` > **Key pattern**: CreateWorkflowDefinition → CreateNode (with ContainerId + outputs.nodeOutputs) → UpdateWorkflowDefinition (add trigger) → **CreatePipelineRun (deploy)**. Each node within a workflow MUST have `outputs.nodeOutputs`. **The workflow is NOT active until deployed via CreatePipelineRun.** > > **Dependency wiring summary**: In `spec.dependencies`, `nodeId` is the **current node's own name** (self-reference, NOT the upstream node), and `depends[].output` is the **upstream node's output** (`projectIdentifier.upstream_node_name`). The `outputs.nodeOutputs[].data` value of the upstream node and the `depends[].output` value of the downstream node must be **character-for-character identical**, otherwise the dependency silently fails. ## Core Workflow ### Environment Discovery (Required Before Creating) **Step 0 — CLI Profile Switch (MUST be the very first action):** Run `aliyun configure list`. If multiple profiles exist, run `aliyun configure switch --profile ` (prefer `dataworks`-named profile, otherwise `default`). **No `aliyun dataworks-public` command may run before this.** > **If credentials are empty or invalid, STOP HERE.** Do not proceed with any API calls. Report the error to the user and instruct them to configure valid credentials outside this session (via `aliyun configure` or environment variables). Do not attempt workarounds such as writing config files manually or using placeholder values. Before creating nodes or workflows, understand the project's existing environment. **It is recommended to use a subagent to execute queries**, returning only a summary to the main Agent to avoid raw data consuming too much context. Subagent tasks: 1. Call `ListWorkflowDefinitions` to get the workflow list 2. Call `ListNodes` to get the existing node list 3. Call `ListDataSources` **AND** `ListComputeResources` to get all available data sources and compute engine bindings (EMR, Hologres, StarRocks, etc.). `ListComputeResources` supplements `ListDataSources` which may not return compute-engine-type resources 4. Return a summary (do not return raw data): - Workflow inventory: name + number of contained nodes + type (scheduled/manual) - Existing nodes relevant to the current task: name + type + parent workflow - Available data sources + compute resources (name, type) — combine both lists - Suggested target workflow (if inferable from the task description) Based on the summary, the main Agent decides: **target workflow** (existing or new, user decides), **node naming** (follow existing conventions), and **dependencies** (infer from SQL references and existing nodes). **Pre-creation conflict check (required, applies to all object types)**: 1. **Name duplication check**: Before creating any object, use the corresponding List API to check if an object with the same name already exists: - Workflow → `ListWorkflowDefinitions` - Node → `ListNodes` (node names are globally unique within a project) - Resource → `ListResources` - Function → `ListFunctions` - Component → `ListComponents` 2. **Handling existing objects**: Inform the user and ask how to proceed (use existing / rename / update existing). **Direct deletion of existing objects is prohibited** 3. **Output name conflict check (CRITICAL)**: A node's `outputs.nodeOutputs[].data` (format `${projectIdentifier}.NodeName`) must be **globally unique within the project**, even across different workflows. Use `ListNodes --Name NodeName` and inspect `Outputs.NodeOutputs[].Data` in the response to verify. If the output name conflicts with an existing node, the conflict **must be resolved before creation** — otherwise deployment will fail with `"can not exported multiple nodes into the same output"` (see troubleshooting.md #11b) **Certainty level determines interaction approach**: - Certain information → Use directly, do not ask the user - Confident inference → Proceed, explain the reasoning in the output - Uncertain information → Must ask the user ### Creating Nodes **Unified workflow**: Whether in OpenAPI Mode or Git Mode, generate the same local file structure. #### Step 1: Create the Node Directory and Three Files One folder = one node, containing three files: ``` my_node/ ├── my_node.spec.json # FlowSpec node definition ├── my_node.sql # Code file (extension based on contentFormat) └── dataworks.properties # Runtime configuration (actual values) ``` **spec.json** — Copy the minimal Spec from `references/nodetypes/{category}/{TYPE}.md`, modify name and path, and use `${spec.xxx}` placeholders to reference values from properties. If the user specifies trigger, dependencies, rerunTimes, etc., add them to the spec as well. **Code file** — Determine the format (sql/shell/python/json/empty) based on the `contentFormat` in the node type documentation; determine the extension based on the `extension` field. **dataworks.properties** — Fill in actual values: ```properties projectIdentifier= spec.datasource.name= spec.runtimeResource.resourceGroup= ``` Do not fill in uncertain values — if omitted, the server automatically uses project defaults. Reference examples: `assets/templates/` #### Step 2: Submit **Default is OpenAPI** (unless the user explicitly says "commit to Git"): 1. Use `build.py` to merge the three files into API input: ```bash python $SKILL/scripts/build.py ./my_node > /tmp/spec.json ``` build.py does three things (no third-party dependencies; if errors occur, refer to the source code to execute manually): - Read `dataworks.properties` → replace `${spec.xxx}` and `${projectIdentifier}` placeholders in spec.json - Read the code file → embed into `script.content` - Output the merged complete JSON 2. Validate the spec before submission: ```bash python $SKILL/scripts/validate.py ./my_node ``` 3. **Pre-submission spec review (MANDATORY)** — Before calling CreateNode, review the merged JSON against this checklist: - [ ] `script.runtime.command` matches the intended node type (check `references/nodetypes/{category}/{TYPE}.md`) - [ ] `datasource` — Required if the node type needs a data source (see the node type doc's `datasourceType` field). Check that `name` matches an existing data source (`ListDataSources`) or compute resource (`ListComputeResources`), and `type` matches the expected engine type (e.g., `odps`, `hologres`, `emr`, `starrocks`). If unsure, omit and let the server use project defaults - [ ] `runtimeResource.resourceGroup` — Check that the value matches an existing resource group (`ListResourceGroups`). If unsure, omit and let the server use project defaults - [ ] `trigger` — For workflow nodes: omit to inherit the workflow schedule; only set when the user explicitly specifies a per-node schedule. For standalone nodes: set if the user specified a schedule - [ ] `outputs.nodeOutputs` — **Required for workflow nodes**. Format: `{"data":"${projectIdentifier}.NodeName","artifactType":"NodeOutput"}`. Verify the output name is globally unique in the project (`ListNodes --Name`) - [ ] `dependencies` — `nodeId` must be the **current node's own name** (self-reference). `depends[].output` must **exactly match** the upstream node's `outputs.nodeOutputs[].data`. **Every workflow node MUST have dependencies**: root nodes (no upstream) MUST depend on `${projectIdentifier}_root` (underscore, not dot); downstream nodes depend on upstream outputs. A workflow node with NO dependencies entry will become an orphan - [ ] No invented fields — Compare against the FlowSpec Anti-Patterns table above; remove any field not documented in `references/flowspec-guide.md` 4. Call the API to submit (refer to [references/api/CreateNode.md](references/api/CreateNode.md)): ```bash # DataWorks 2024-05-18 API does not yet have plugin mode (kebab-case), use RPC direct invocation format (PascalCase) aliyun dataworks-public CreateNode \ --ProjectId $PROJECT_ID \ --Scene DATAWORKS_PROJECT \ --Spec "$(cat /tmp/spec.json)" \ --user-agent AlibabaCloud-Agent-Skills ``` > **Note**: `aliyun dataworks-public CreateNode` is in RPC direct invocation format and **does not require any plugin installation**. If the command is not found, check the aliyun CLI version (requires >= 3.3.1). **Never** downgrade to legacy kebab-case commands (`create-file`/`create-folder`). > **Sandbox fallback**: If `$(cat ...)` is blocked, use Python `subprocess.run(['aliyun', 'dataworks-public', 'CreateNode', '--ProjectId', str(PID), '--Scene', 'DATAWORKS_PROJECT', '--Spec', spec_str, '--user-agent', 'AlibabaCloud-Agent-Skills'])`. 5. To place within a workflow, add `--ContainerId $WorkflowId` **Git Mode** (when the user explicitly requests): `git add ./my_node && git commit`, DataWorks automatically syncs and replaces placeholders **Minimum required fields** (verified in practice, universal across all 130+ types): - `name` — Node name - `id` — **Must be set equal to `name`**. Ensures `spec.dependencies[*].nodeId` can match. Without explicit `id`, the API may silently drop dependencies - `script.path` — Script path, must end with the node name; the server automatically prepends the workflow prefix - `script.runtime.command` — Node type (e.g., ODPS_SQL, DIDE_SHELL) **Copyable minimal node Spec** (Shell node example): ```json {"version":"2.0.0","kind":"Node","spec":{"nodes":[{ "name":"my_shell_node","id":"my_shell_node", "script":{"path":"my_shell_node","runtime":{"command":"DIDE_SHELL"},"content":"#!/bin/bash\necho hello"} }]}} ``` Other fields are not required; the server will automatically fill in project defaults: - **datasource, runtimeResource** — If unsure, do not pass them; the server automatically binds project defaults - **trigger** — If not passed, inherits the workflow schedule. Only pass when specified by the user - **dependencies, rerunTimes, etc.** — Only pass when specified by the user - **outputs.nodeOutputs** — Optional for standalone nodes; **required for nodes within a workflow** (`{"data":"${projectIdentifier}.NodeName","artifactType":"NodeOutput"}`), otherwise downstream dependencies silently fail. ⚠️ The output name (`${projectIdentifier}.NodeName`) must be **globally unique within the project** — if another node (even in a different workflow) already uses the same output name, deployment will fail with "can not exported multiple nodes into the same output". Always check with `ListNodes` before creating ### Workflow and Node Relationship ``` Project └── Workflow ← Container, unified scheduling management ├── Node A ← Minimum execution unit ├── Node B (depends A) └── Node C (depends B) ``` - A **workflow** is the container and scheduling unit for nodes, with its own trigger and strategy - **Nodes** can exist independently at the root level or belong to a workflow (user decides) - The workflow's `script.runtime.command` is always `"WORKFLOW"` - Dependency configuration for nodes within a workflow: only maintain dependencies in the `spec.dependencies` array (do NOT dual-write `inputs.nodeOutputs`). ⚠️ `spec.dependencies[*].nodeId` is a **self-reference** — it must match the **current node's own `name`** (the node that HAS the dependency), NOT the upstream node's name or ID. `depends[].output` is the **upstream node's output identifier** (`${projectIdentifier}.UpstreamNodeName`). Upstream nodes must declare `outputs.nodeOutputs` ### Creating Workflows 1. **Create the workflow definition** (minimal spec): ```json {"version":"2.0.0","kind":"CycleWorkflow","spec":{"workflows":[{ "name":"workflow_name","script":{"path":"workflow_name","runtime":{"command":"WORKFLOW"}} }]}} ``` Call `CreateWorkflowDefinition` → returns WorkflowId 2. **Create nodes in dependency order** (each node passes `ContainerId=WorkflowId`) - **Before each node**: Check that `${projectIdentifier}.NodeName` is not already used as an output by any existing node in the project (use `ListNodes` with `--Name` and inspect `Outputs.NodeOutputs[].Data`). Duplicate output names cause deployment failure - Each node's spec **must include** `outputs.nodeOutputs`: `{"data":"${projectIdentifier}.NodeName","artifactType":"NodeOutput"}` - Downstream nodes declare dependencies in `spec.dependencies`: `nodeId` = **current node's own name** (self-reference), `depends[].output` = **upstream node's output** (see workflow-guide.md) 3. **Verify dependencies (MANDATORY after all nodes created)** — For each downstream node, call `ListNodeDependencies --Id `. If `TotalCount` is `0` but the node should have upstream dependencies, the CreateNode API silently dropped them. **Fix immediately** with `UpdateNode` using `spec.dependencies` (see "Updating dependencies" below). Do NOT proceed to deploy until all dependencies are confirmed 4. **Set the schedule** — `UpdateWorkflowDefinition` with `trigger` (if the user specified a schedule) 5. **Deploy online (REQUIRED)** — `CreatePipelineRun(Type=Online, ObjectIds=[WorkflowId])` → poll `GetPipelineRun` → advance stages with `ExecPipelineRunStage`. **A workflow is NOT active until deployed.** Do not skip this step or tell the user to do it manually. Detailed guide and copyable complete node Spec examples (including outputs and dependencies): [references/workflow-guide.md](references/workflow-guide.md) ### Updating Existing Nodes **Must use incremental updates** — only pass the node id + fields to modify: ```json {"version":"2.0.0","kind":"Node","spec":{"nodes":[{ "id":"NodeID", "script":{"content":"new code"} }]}} ``` > **⚠️ Critical**: UpdateNode **always** uses `"kind":"Node"`, even if the node belongs to a workflow. Do NOT use `"kind":"CycleWorkflow"` — that is only for workflow-level operations (`UpdateWorkflowDefinition`). **Do not pass unchanged fields like datasource or runtimeResource** (the server may have corrected values; passing them back can cause errors). > **⚠️ Updating dependencies**: To fix or change a node's dependencies via UpdateNode, use `spec.dependencies` — **NEVER use `inputs.nodeOutputs`**. Example: > ```json > {"version":"2.0.0","kind":"Node","spec":{"nodes":[{"id":"NodeID"}],"dependencies":[{"nodeId":"current_node_name","depends":[{"type":"Normal","output":"project.upstream_node"}]}]}} > ``` #### Update + Republish Workflow Complete end-to-end flow for modifying an existing node and deploying the change: 1. **Find the node** — `ListNodes(Name=xxx)` → get Node ID 2. **Update the node** — `UpdateNode` with incremental spec (`kind:Node`, only `id` + changed fields) 3. **Publish** — `CreatePipelineRun(type=Online, object_ids=[NodeID])` → poll `GetPipelineRun` → advance stages with `ExecPipelineRunStage` ```bash # Step 1: Find the node aliyun dataworks-public ListNodes --ProjectId $PID --Name "my_node" --user-agent AlibabaCloud-Agent-Skills # → Note the node Id from the response # Step 2: Update (incremental — only id + changed fields) aliyun dataworks-public UpdateNode --ProjectId $PID --Id $NODE_ID \ --Spec '{"version":"2.0.0","kind":"Node","spec":{"nodes":[{"id":"'$NODE_ID'","script":{"content":"SELECT 1;"}}]}}' \ --user-agent AlibabaCloud-Agent-Skills # Step 3: Publish (see "Publishing and Deploying" below) aliyun dataworks-public CreatePipelineRun --ProjectId $PID \ --PipelineRunParam '{"type":"Online","objectIds":["'$NODE_ID'"]}' \ --user-agent AlibabaCloud-Agent-Skills ``` > **Common wrong paths after UpdateNode** (all prohibited): > - ❌ `DeployFile` / `SubmitFile` — legacy APIs, will fail or behave unexpectedly > - ❌ `ImportWorkflowDefinition` — for initial bulk import only, not for updating or publishing > - ❌ `ListFiles` / `GetFile` — legacy file model, use `ListNodes` / `GetNode` instead > - ✅ `CreatePipelineRun` → `GetPipelineRun` → `ExecPipelineRunStage` ### Publishing and Deploying > **⚠️ NEVER use `DeployFile`, `SubmitFile`, `ListDeploymentPackages`, `GetDeploymentPackage`, `ListFiles`, or `GetFile` for deployment.** These are all legacy APIs. Use ONLY: `CreatePipelineRun` → `GetPipelineRun` → `ExecPipelineRunStage`. Publishing is an asynchronous multi-stage pipeline: 1. `CreatePipelineRun(Type=Online, ObjectIds=[ID])` → get PipelineRunId 2. Poll `GetPipelineRun` → check `Pipeline.Status` and `Pipeline.Stages` 3. When a Stage has `Init` status and all preceding Stages are `Success` → call `ExecPipelineRunStage(Code=Stage.Code)` to advance 4. Until the Pipeline overall status becomes `Success` / `Fail` **Key point**: The Build stage runs automatically, but the Check and Deploy stages must be manually advanced. Detailed CLI examples and polling scripts are in [references/deploy-guide.md](references/deploy-guide.md). > **CLI Note**: The `aliyun` CLI returns JSON with the top-level key `Pipeline` (not SDK's `resp.body.pipeline`); Stages are in `Pipeline.Stages`. ## Common Node Types | Use Case | command | contentFormat | Extension | datasource | |------|---------|--------------|------|------------| | Shell script | DIDE_SHELL | shell | .sh | — | | MaxCompute SQL | ODPS_SQL | sql | .sql | odps | | Python script | PYTHON | python | .py | — | | Offline data sync | DI | json | .json | — | | Hologres SQL | HOLOGRES_SQL | sql | .sql | hologres | | Flink streaming SQL | FLINK_SQL_STREAM | sql | .json | flink | | Flink batch SQL | FLINK_SQL_BATCH | sql | .json | flink | | EMR Hive | EMR_HIVE | sql | .sql | emr | | EMR Spark SQL | EMR_SPARK_SQL | sql | .sql | emr | | Serverless Spark SQL | SERVERLESS_SPARK_SQL | sql | .sql | emr | | StarRocks SQL | StarRocks | sql | .sql | starrocks | | ClickHouse SQL | CLICK_SQL | sql | .sql | clickhouse | | Virtual node | VIRTUAL | empty | .vi | — | Complete list (130+ types): [references/nodetypes/index.md](references/nodetypes/index.md) (searchable by command name, description, and category, with links to detailed documentation for each type) **When you cannot find a node type**: 1. Check `references/nodetypes/index.md` and match by keyword 2. `Glob("**/{keyword}*.md", path="references/nodetypes")` to locate the documentation directly 3. Use the `GetNode` API to get the spec of a similar node from the live environment as a reference 4. If none of the above works → fall back to `DIDE_SHELL` and use command-line tools within the Shell to accomplish the task ## Key Constraints 1. **script.path is required**: Script path, must end with the node name. When creating, you can pass just the node name; the server automatically prepends the workflow prefix 2. **Dependencies are configured via `spec.dependencies`** (do NOT dual-write `inputs.nodeOutputs`): In `spec.dependencies`, `nodeId` is a **self-reference** — it must be the **current node's own `name`** (the node being created), NOT the upstream node. `depends[].output` is the **upstream node's output** (`${projectIdentifier}.UpstreamNodeName`). The upstream's `outputs.nodeOutputs[].data` and downstream's `depends[].output` must be **character-for-character identical**. Upstream nodes must declare `outputs.nodeOutputs`. ⚠️ Output names (`${projectIdentifier}.NodeName`) must be **globally unique within the project** — duplicates cause deployment failure 3. **Immutable properties**: A node's `command` (node type) cannot be changed after creation; if incorrect, inform the user and suggest creating a new node with the correct type 4. **Updates must be incremental**: Only pass id + fields to modify; do not pass unchanged fields like datasource/runtimeResource 5. **datasource.type may be corrected by the server**: e.g., `flink` → `flink_serverless`; use the generic type when creating 6. **Nodes can exist independently**: Nodes can be created at the root level (without passing ContainerId) or belong to a workflow (pass ContainerId=WorkflowId). Whether to place in a workflow is the user's decision 7. **Workflow command is always WORKFLOW**: `script.runtime.command` must be `"WORKFLOW"` 8. **Deletion is not supported by this skill**: This skill does not provide any delete operations. When creation or publishing fails, **never** attempt to "fix" the problem by deleting existing objects. Correct approach: diagnose the failure cause → inform the user of the specific conflict → let the user decide how to handle it (rename / update existing) 9. **Name conflict check is required before creation**: Before calling any Create API, use the corresponding List API to confirm the name is not duplicated (see "Environment Discovery"). Name conflicts will cause creation failure; duplicate node output names (`outputs.nodeOutputs[].data`) will cause dependency errors or publishing failure 10. **Mutating operations require user confirmation**: Except for Create and read-only queries (Get/List), all OpenAPI operations that modify existing objects (Update, Move, Rename, etc.) **must be shown to the user with explicit confirmation obtained before execution**. Confirmation information should include: operation type, target object name/ID, and key changes. These APIs must not be called before user confirmation. **Delete and Abolish operations are not supported by this skill** 11. **Use only 2024-05-18 version APIs**: All APIs in this skill are DataWorks 2024-05-18 version. Legacy APIs (`create-file`, `create-folder`, `CreateFlowProject`, etc.) are prohibited. If an API call returns an error, first check [troubleshooting.md](references/troubleshooting.md); do not fall back to legacy APIs 12. **Stop on errors instead of brute-force retrying**: If the same error code appears more than 2 consecutive times, the approach is wrong. Stop and analyze the error cause (check [troubleshooting.md](references/troubleshooting.md)) instead of repeatedly retrying the same incorrect API with different parameters. **Never fall back to legacy APIs** (`create-file`, `create-business`, etc.) when a new API fails — review the FlowSpec Anti-Patterns table at the top of this document instead. **Specific trap**: If `aliyun help` output mentions "Plugin available but not installed" for dataworks-public, do NOT install the plugin — this leads to using deprecated kebab-case APIs. Instead, use PascalCase RPC directly (e.g., `aliyun dataworks-public CreateNode`) 13. **CLI parameter names must be checked in documentation, guessing is prohibited**: Before calling an API, you must first check `references/api/{APIName}.md` to confirm parameter names. Common mistakes: `GetProject`'s ID parameter is `--Id` (not `--ProjectId`); `UpdateNode` requires `--Id`. When unsure, verify with `aliyun dataworks-public {APIName} --help` 14. **PascalCase RPC only, no kebab-case**: CLI commands must use `aliyun dataworks-public CreateNode` (PascalCase), never `aliyun dataworks-public create-node` (kebab-case). No plugin installation is needed. If the command is not found, upgrade `aliyun` CLI to >= 3.3.1 15. **No wrapper scripts**: Run each `aliyun` CLI command directly in the shell. Never create `.sh`/`.py` wrapper scripts to batch multiple API calls — this obscures errors and makes debugging impossible. Execute one API call at a time, check the response, then proceed 16. **API response = success, not file output**: Writing JSON spec files to disk is a preparation step, not completion. The task is complete only when the `aliyun` CLI returns a success response with a valid `Id`. If the API call fails, fix the spec and retry — do not declare the task done by saving local files 17. **On error: re-read the Quick Start, do not invent new approaches**: When an API call fails, compare your spec against the exact Quick Start example at the top of this document field by field. The most common cause is an invented FlowSpec field that does not exist. Copy the working example and modify only the values you need to change 18. **Idempotency protection for write operations**: DataWorks 2024-05-18 Create APIs (`CreateNode`, `CreateWorkflowDefinition`, `CreatePipelineRun`, etc.) do not support a `ClientToken` parameter. To prevent duplicate resource creation on network retries or timeouts: - **Before creating**: Always run the pre-creation conflict check (List API) as described in "Environment Discovery" — this is the primary idempotency gate - **After a network error or timeout on Create**: Do NOT blindly retry. First call the corresponding List/Get API to check whether the resource was actually created (the server may have processed the request despite the client-side error). Only retry if the resource does not exist - **Record RequestId**: Every API response includes a `RequestId` field. Log it so that duplicate-creation incidents can be traced and resolved via Alibaba Cloud support ## API Quick Reference > **API Version**: All APIs listed below are DataWorks **2024-05-18** version. CLI invocation format: `aliyun dataworks-public {APIName} --Parameter --user-agent AlibabaCloud-Agent-Skills` (PascalCase RPC direct invocation; DataWorks 2024-05-18 does not yet have plugin mode). **Only use the APIs listed in the table below**; do not search for or use other DataWorks APIs. Detailed parameters and code templates for each API are in `references/api/{APIName}.md`. If a call returns an error, you can get the latest definition from `https://api.aliyun.com/meta/v1/products/dataworks-public/versions/2024-05-18/apis/{APIName}/api.json`. ### Components | API | Description | |-----|------| | [CreateComponent](references/api/CreateComponent.md) | Create a component | | [GetComponent](references/api/GetComponent.md) | Get component details | | [UpdateComponent](references/api/UpdateComponent.md) | Update a component | | [ListComponents](references/api/ListComponents.md) | List components | ### Nodes | API | Description | |-----|------| | [CreateNode](references/api/CreateNode.md) | Create a data development node. project_id + scene + spec, optional container_id | | [UpdateNode](references/api/UpdateNode.md) | Update node information. Incremental update, only pass id + fields to change | | [MoveNode](references/api/MoveNode.md) | Move a node to a specified path | | [RenameNode](references/api/RenameNode.md) | Rename a node | | [GetNode](references/api/GetNode.md) | Get node details, returns the complete spec | | [ListNodes](references/api/ListNodes.md) | List nodes, supports filtering by workflow | | [ListNodeDependencies](references/api/ListNodeDependencies.md) | List a node's dependency nodes | ### Workflow Definitions | API | Description | |-----|------| | [CreateWorkflowDefinition](references/api/CreateWorkflowDefinition.md) | Create a workflow. project_id + spec | | [ImportWorkflowDefinition](references/api/ImportWorkflowDefinition.md) | Import a workflow (initial bulk import ONLY — do NOT use for updates or publishing; use `UpdateNode` + `CreatePipelineRun` instead) | | [UpdateWorkflowDefinition](references/api/UpdateWorkflowDefinition.md) | Update workflow information, incremental update | | [MoveWorkflowDefinition](references/api/MoveWorkflowDefinition.md) | Move a workflow to a target path | | [RenameWorkflowDefinition](references/api/RenameWorkflowDefinition.md) | Rename a workflow | | [GetWorkflowDefinition](references/api/GetWorkflowDefinition.md) | Get workflow details | | [ListWorkflowDefinitions](references/api/ListWorkflowDefinitions.md) | List workflows, filter by type | ### Resources | API | Description | |-----|------| | [CreateResource](references/api/CreateResource.md) | Create a file resource | | [UpdateResource](references/api/UpdateResource.md) | Update file resource information, incremental update | | [MoveResource](references/api/MoveResource.md) | Move a file resource to a specified directory | | [RenameResource](references/api/RenameResource.md) | Rename a file resource | | [GetResource](references/api/GetResource.md) | Get file resource details | | [ListResources](references/api/ListResources.md) | List file resources | ### Functions | API | Description | |-----|------| | [CreateFunction](references/api/CreateFunction.md) | Create a UDF function | | [UpdateFunction](references/api/UpdateFunction.md) | Update UDF function information, incremental update | | [MoveFunction](references/api/MoveFunction.md) | Move a function to a target path | | [RenameFunction](references/api/RenameFunction.md) | Rename a function | | [GetFunction](references/api/GetFunction.md) | Get function details | | [ListFunctions](references/api/ListFunctions.md) | List functions | ### Publishing Pipeline | API | Description | |-----|------| | [CreatePipelineRun](references/api/CreatePipelineRun.md) | Create a publishing pipeline. type=Online/Offline | | [ExecPipelineRunStage](references/api/ExecPipelineRunStage.md) | Execute a specified stage of the publishing pipeline, async requires polling | | [GetPipelineRun](references/api/GetPipelineRun.md) | Get publishing pipeline details, returns Stages status | | [ListPipelineRuns](references/api/ListPipelineRuns.md) | List publishing pipelines | | [ListPipelineRunItems](references/api/ListPipelineRunItems.md) | Get publishing content | ### Auxiliary Queries | API | Description | |-----|------| | [GetProject](references/api/GetProject.md) | Get projectIdentifier by id | | [ListDataSources](references/api/ListDataSources.md) | List data sources | | [ListComputeResources](references/api/ListComputeResources.md) | List compute engine bindings (EMR, Hologres, StarRocks, etc.) — supplements ListDataSources | | [ListResourceGroups](references/api/ListResourceGroups.md) | List resource groups | ## Reference Documentation | Scenario | Document | |------|------| | Complete list of APIs and CLI commands | [references/related-apis.md](references/related-apis.md) | | RAM permission policy configuration | [references/ram-policies.md](references/ram-policies.md) | | Operation verification methods | [references/verification-method.md](references/verification-method.md) | | Acceptance criteria and test cases | [references/acceptance-criteria.md](references/acceptance-criteria.md) | | CLI installation and configuration guide | [references/cli-installation-guide.md](references/cli-installation-guide.md) | | Node type index (130+ types) | [references/nodetypes/index.md](references/nodetypes/index.md) | | FlowSpec field reference | [references/flowspec-guide.md](references/flowspec-guide.md) | | Workflow development | [references/workflow-guide.md](references/workflow-guide.md) | | Scheduling configuration | [references/scheduling-guide.md](references/scheduling-guide.md) | | Publishing and unpublishing | [references/deploy-guide.md](references/deploy-guide.md) | | DI data integration | [references/di-guide.md](references/di-guide.md) | | Troubleshooting | [references/troubleshooting.md](references/troubleshooting.md) | | Complete examples | [assets/templates/README.md](assets/templates/README.md) | FILE:assets/templates/01-shell-node/README.md # Example 01: Shell Node The simplest DataWorks node example. Creates a Shell script node that runs daily at midnight. ## File Structure ``` hello/ ├── hello.spec.json # Node definition ├── hello.sh # Shell script └── dataworks.properties # Configuration ``` ## Creation Steps 1. Create a spec file based on hello/hello.spec.json 2. Modify name, path, and content 3. Write hello.sh 4. Create dataworks.properties 5. Git Mode: `git add && git commit` 6. OpenAPI Mode: Call CreateNode API directly with minSpec FILE:assets/templates/01-shell-node/hello/hello.sh #!/bin/bash echo "Hello DataWorks!" echo "Business date: $bizdate" date FILE:assets/templates/01-shell-node/hello/hello.spec.json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "hello", "id": "hello", "recurrence": "Normal", "timeout": 4, "timeoutUnit": "HOURS", "instanceMode": "T+1", "rerunMode": "Allowed", "rerunTimes": 0, "rerunInterval": 180000, "script": { "path": "hello", "language": "shell", "runtime": { "command": "DIDE_SHELL" }, "content": "", "parameters": [ { "name": "bizdate", "scope": "NodeParameter", "type": "System", "value": "$yyyymmdd", "artifactType": "Variable" } ] }, "trigger": { "type": "Scheduler", "cron": "00 00 00 * * ?", "startTime": "1970-01-01 00:00:00", "endTime": "9999-01-01 00:00:00", "timezone": "Asia/Shanghai" }, "runtimeResource": { "resourceGroup": "${spec.runtimeResource.resourceGroup}" }, "outputs": { "nodeOutputs": [ { "data": "${projectIdentifier}.hello", "artifactType": "NodeOutput" } ] }, "inputs": { "nodeOutputs": [ { "data": "${projectIdentifier}_root", "artifactType": "NodeOutput" } ] } } ], "dependencies": [ { "nodeId": "hello", "depends": [ { "type": "Normal", "output": "${projectIdentifier}_root" } ] } ] } } FILE:assets/templates/02-odps-sql-node/README.md # Example 02: MaxCompute SQL Node SQL node with datasource. Demonstrates datasource configuration. ## File Structure ``` dwd_user_info/ ├── dwd_user_info.spec.json ├── dwd_user_info.sql └── dataworks.properties ``` FILE:assets/templates/02-odps-sql-node/dwd_user_info/dwd_user_info.spec.json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "dwd_user_info", "id": "dwd_user_info", "recurrence": "Normal", "timeout": 4, "timeoutUnit": "HOURS", "instanceMode": "T+1", "rerunMode": "Allowed", "rerunTimes": 0, "rerunInterval": 180000, "script": { "path": "dwd_user_info", "language": "odps-sql", "runtime": { "command": "ODPS_SQL" }, "content": "", "parameters": [ { "name": "bizdate", "scope": "NodeParameter", "type": "System", "value": "$yyyymmdd", "artifactType": "Variable" } ] }, "trigger": { "type": "Scheduler", "cron": "00 00 02 * * ?", "startTime": "1970-01-01 00:00:00", "endTime": "9999-01-01 00:00:00", "timezone": "Asia/Shanghai" }, "datasource": { "name": "${spec.datasource.name}", "type": "odps" }, "runtimeResource": { "resourceGroup": "${spec.runtimeResource.resourceGroup}" }, "outputs": { "nodeOutputs": [ { "data": "${projectIdentifier}.dwd_user_info", "artifactType": "NodeOutput" } ] }, "inputs": { "nodeOutputs": [ { "data": "${projectIdentifier}_root", "artifactType": "NodeOutput" } ] } } ], "dependencies": [ { "nodeId": "dwd_user_info", "depends": [ { "type": "Normal", "output": "${projectIdentifier}_root" } ] } ] } } FILE:assets/templates/02-odps-sql-node/dwd_user_info/dwd_user_info.sql -- DWD user info table -- Processed from ODS layer daily at 2:00 AM INSERT OVERWRITE TABLE dwd_user_info PARTITION (dt='${bizdate}') SELECT user_id, user_name, age, gender, city, register_time, CURRENT_TIMESTAMP AS etl_time FROM ods_user_info WHERE dt = '${bizdate}'; FILE:assets/templates/03-sql-with-dependency/README.md # Example 03: SQL Inter-Node Dependency Two ODPS SQL nodes where dwd_order_detail depends on ods_order. ## File Structure ``` ods_order/ ├── ods_order.spec.json ├── ods_order.sql └── dataworks.properties dwd_order_detail/ ├── dwd_order_detail.spec.json ├── dwd_order_detail.sql └── dataworks.properties ``` FILE:assets/templates/03-sql-with-dependency/dwd_order_detail/dwd_order_detail.spec.json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "dwd_order_detail", "id": "dwd_order_detail", "recurrence": "Normal", "timeout": 4, "timeoutUnit": "HOURS", "instanceMode": "T+1", "rerunMode": "Allowed", "rerunTimes": 0, "rerunInterval": 180000, "script": { "path": "dwd_order_detail", "language": "odps-sql", "runtime": { "command": "ODPS_SQL" }, "content": "", "parameters": [ { "name": "bizdate", "scope": "NodeParameter", "type": "System", "value": "$yyyymmdd", "artifactType": "Variable" } ] }, "trigger": { "type": "Scheduler", "cron": "00 00 02 * * ?", "startTime": "1970-01-01 00:00:00", "endTime": "9999-01-01 00:00:00", "timezone": "Asia/Shanghai" }, "datasource": { "name": "${spec.datasource.name}", "type": "odps" }, "runtimeResource": { "resourceGroup": "${spec.runtimeResource.resourceGroup}" }, "outputs": { "nodeOutputs": [ { "data": "${projectIdentifier}.dwd_order_detail", "artifactType": "NodeOutput" } ] }, "inputs": { "nodeOutputs": [ { "data": "${projectIdentifier}.ods_order", "artifactType": "NodeOutput" } ] } } ], "dependencies": [ { "nodeId": "dwd_order_detail", "depends": [ { "type": "Normal", "output": "${projectIdentifier}.ods_order" } ] } ] } } FILE:assets/templates/03-sql-with-dependency/dwd_order_detail/dwd_order_detail.sql -- DWD order detail table -- Runs after ods_order completes INSERT OVERWRITE TABLE dwd_order_detail PARTITION (dt='${bizdate}') SELECT o.order_id, o.user_id, u.user_name, o.product_id, p.product_name, o.amount, o.order_time FROM ods_order o LEFT JOIN dim_user u ON o.user_id = u.user_id LEFT JOIN dim_product p ON o.product_id = p.product_id WHERE o.dt = '${bizdate}'; FILE:assets/templates/03-sql-with-dependency/ods_order/ods_order.spec.json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "ods_order", "id": "ods_order", "recurrence": "Normal", "timeout": 4, "timeoutUnit": "HOURS", "instanceMode": "T+1", "rerunMode": "Allowed", "rerunTimes": 0, "rerunInterval": 180000, "script": { "path": "ods_order", "language": "odps-sql", "runtime": { "command": "ODPS_SQL" }, "content": "", "parameters": [ { "name": "bizdate", "scope": "NodeParameter", "type": "System", "value": "$yyyymmdd", "artifactType": "Variable" } ] }, "trigger": { "type": "Scheduler", "cron": "00 00 01 * * ?", "startTime": "1970-01-01 00:00:00", "endTime": "9999-01-01 00:00:00", "timezone": "Asia/Shanghai" }, "datasource": { "name": "${spec.datasource.name}", "type": "odps" }, "runtimeResource": { "resourceGroup": "${spec.runtimeResource.resourceGroup}" }, "outputs": { "nodeOutputs": [ { "data": "${projectIdentifier}.ods_order", "artifactType": "NodeOutput" } ] }, "inputs": { "nodeOutputs": [ { "data": "${projectIdentifier}_root", "artifactType": "NodeOutput" } ] } } ], "dependencies": [ { "nodeId": "ods_order", "depends": [ { "type": "Normal", "output": "${projectIdentifier}_root" } ] } ] } } FILE:assets/templates/03-sql-with-dependency/ods_order/ods_order.sql -- ODS order table INSERT OVERWRITE TABLE ods_order PARTITION (dt='${bizdate}') SELECT order_id, user_id, product_id, amount, order_time FROM raw_order WHERE dt = '${bizdate}'; FILE:assets/templates/04-di-mysql-to-maxcompute/README.md # Example 04: DI Data Synchronization (MySQL to MaxCompute) Offline data synchronization node that reads data from MySQL and writes to MaxCompute. ## File Structure ``` sync_user/ ├── sync_user.spec.json ├── sync_user.json # DI task code (JSON format) └── dataworks.properties ``` FILE:assets/templates/04-di-mysql-to-maxcompute/sync_user/sync_user.json { "type": "job", "version": "2.0", "steps": [ { "stepType": "mysql", "name": "Reader", "category": "reader", "parameter": { "datasource": "mysql_source", "column": ["id", "user_name", "age", "gender", "city", "register_time"], "connection": [ { "table": ["user_info"], "datasource": "mysql_source" } ], "where": "dt = '${bizdate}'", "splitPk": "id", "encoding": "UTF-8" } }, { "stepType": "odps", "name": "Writer", "category": "writer", "parameter": { "datasource": "odps_target", "table": "ods_user_info", "column": [ {"name": "id", "type": "bigint"}, {"name": "user_name", "type": "string"}, {"name": "age", "type": "bigint"}, {"name": "gender", "type": "string"}, {"name": "city", "type": "string"}, {"name": "register_time", "type": "datetime"} ], "partition": "dt=${bizdate}", "truncate": true } } ], "order": { "hops": [ { "from": "Reader", "to": "Writer" } ] }, "setting": { "speed": { "concurrent": 3, "throttle": false }, "errorLimit": { "record": 0 } } } FILE:assets/templates/04-di-mysql-to-maxcompute/sync_user/sync_user.spec.json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "sync_user", "id": "sync_user", "recurrence": "Normal", "timeout": 4, "timeoutUnit": "HOURS", "instanceMode": "T+1", "rerunMode": "Allowed", "rerunTimes": 0, "rerunInterval": 180000, "script": { "path": "sync_user", "language": "di", "runtime": { "command": "DI" }, "content": "", "parameters": [ { "name": "bizdate", "scope": "NodeParameter", "type": "System", "value": "$yyyymmdd", "artifactType": "Variable" } ] }, "trigger": { "type": "Scheduler", "cron": "00 00 01 * * ?", "startTime": "1970-01-01 00:00:00", "endTime": "9999-01-01 00:00:00", "timezone": "Asia/Shanghai" }, "runtimeResource": { "resourceGroup": "${spec.runtimeResource.resourceGroup}" }, "outputs": { "nodeOutputs": [ { "data": "${projectIdentifier}.sync_user", "artifactType": "NodeOutput" } ] }, "inputs": { "nodeOutputs": [ { "data": "${projectIdentifier}_root", "artifactType": "NodeOutput" } ] } } ], "dependencies": [ { "nodeId": "sync_user", "depends": [ { "type": "Normal", "output": "${projectIdentifier}_root" } ] } ] } } FILE:assets/templates/05-cycle-workflow/README.md # Example 05: Scheduled Workflow ETL workflow with 3 child nodes: extract (Shell) -> transform (SQL) -> load (SQL). ## File Structure ``` my_etl/ ├── my_etl.spec.json # Workflow definition ├── dataworks.properties ├── extract/ │ ├── extract.spec.json │ ├── extract.sh │ └── dataworks.properties ├── transform/ │ ├── transform.spec.json │ ├── transform.sql │ └── dataworks.properties └── load/ ├── load.spec.json ├── load.sql └── dataworks.properties ``` ## Deployment Order 1. Create workflow -> obtain WorkflowId 2. Create extract node (ContainerId=WorkflowId) 3. Create transform node (ContainerId=WorkflowId) 4. Create load node (ContainerId=WorkflowId) 5. Publish and go live FILE:assets/templates/05-cycle-workflow/_deploy.md # Deployment Command Sequence ```bash PROJECT_ID=123456 # 1. Create workflow # Note: Workflow spec must include script.runtime.command="WORKFLOW" # Build spec JSON and call API aliyun dataworks-public CreateWorkflowDefinition \ --ProjectId $PROJECT_ID \ --Spec "$(cat /tmp/wf.json)" # -> Record the returned WorkflowId WF_ID="" # 2. Create extract node (root node, no upstream dependency) # Build spec JSON and call API aliyun dataworks-public CreateNode \ --ProjectId $PROJECT_ID \ --Scene DATAWORKS_PROJECT \ --ContainerId $WF_ID \ --Spec "$(cat /tmp/n1.json)" # 3. Create transform node (depends on extract) # Note: Dependencies are configured via spec.dependencies; ensure dependencies[*].nodeId exactly matches the node id # Build spec JSON and call API aliyun dataworks-public CreateNode \ --ProjectId $PROJECT_ID \ --Scene DATAWORKS_PROJECT \ --ContainerId $WF_ID \ --Spec "$(cat /tmp/n2.json)" # 4. Create load node (depends on transform) # Build spec JSON and call API aliyun dataworks-public CreateNode \ --ProjectId $PROJECT_ID \ --Scene DATAWORKS_PROJECT \ --ContainerId $WF_ID \ --Spec "$(cat /tmp/n3.json)" # 5. Publish and go live aliyun dataworks-public CreatePipelineRun \ --ProjectId $PROJECT_ID \ --Type Online \ --ObjectIds "[\"$WF_ID\"]" ``` FILE:assets/templates/05-cycle-workflow/my_etl/extract/extract.sh #!/bin/bash # Extract: Pull data from source system echo "Extracting data for date: $bizdate" # Actual logic: Call datasource API or perform data extraction echo "Extract completed." FILE:assets/templates/05-cycle-workflow/my_etl/extract/extract.spec.json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "extract", "id": "extract", "recurrence": "Normal", "timeout": 2, "timeoutUnit": "HOURS", "instanceMode": "T+1", "rerunMode": "Allowed", "rerunTimes": 3, "rerunInterval": 180000, "script": { "path": "extract", "language": "shell", "runtime": { "command": "DIDE_SHELL" }, "content": "", "parameters": [ { "name": "bizdate", "scope": "NodeParameter", "type": "System", "value": "$yyyymmdd", "artifactType": "Variable" } ] }, "trigger": { "type": "Scheduler", "cron": "00 00 00 * * ?", "startTime": "1970-01-01 00:00:00", "endTime": "9999-01-01 00:00:00", "timezone": "Asia/Shanghai" }, "runtimeResource": { "resourceGroup": "${spec.runtimeResource.resourceGroup}" }, "outputs": { "nodeOutputs": [ { "data": "${projectIdentifier}.extract", "artifactType": "NodeOutput" } ] }, "inputs": { "nodeOutputs": [ { "data": "${projectIdentifier}_root", "artifactType": "NodeOutput" } ] } } ], "dependencies": [ { "nodeId": "extract", "depends": [ { "type": "Normal", "output": "${projectIdentifier}_root" } ] } ] } } FILE:assets/templates/05-cycle-workflow/my_etl/load/load.spec.json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "load", "id": "load", "recurrence": "Normal", "timeout": 4, "timeoutUnit": "HOURS", "instanceMode": "T+1", "rerunMode": "Allowed", "rerunTimes": 0, "rerunInterval": 180000, "script": { "path": "load", "language": "odps-sql", "runtime": { "command": "ODPS_SQL" }, "content": "", "parameters": [ { "name": "bizdate", "scope": "NodeParameter", "type": "System", "value": "$yyyymmdd", "artifactType": "Variable" } ] }, "trigger": { "type": "Scheduler", "cron": "00 00 00 * * ?", "startTime": "1970-01-01 00:00:00", "endTime": "9999-01-01 00:00:00", "timezone": "Asia/Shanghai" }, "datasource": { "name": "${spec.datasource.name}", "type": "odps" }, "runtimeResource": { "resourceGroup": "${spec.runtimeResource.resourceGroup}" }, "outputs": { "nodeOutputs": [ { "data": "${projectIdentifier}.load", "artifactType": "NodeOutput" } ] }, "inputs": { "nodeOutputs": [ { "data": "${projectIdentifier}.transform", "artifactType": "NodeOutput" } ] } } ], "dependencies": [ { "nodeId": "load", "depends": [ { "type": "Normal", "output": "${projectIdentifier}.transform" } ] } ] } } FILE:assets/templates/05-cycle-workflow/my_etl/load/load.sql -- Load: Write aggregated results to ADS layer INSERT OVERWRITE TABLE ads_order_summary PARTITION (dt='${bizdate}') SELECT user_id, COUNT(*) AS order_count, SUM(amount) AS total_amount, AVG(amount) AS avg_amount FROM dwd_order WHERE dt = '${bizdate}' GROUP BY user_id; FILE:assets/templates/05-cycle-workflow/my_etl/my_etl.spec.json { "version": "2.0.0", "kind": "CycleWorkflow", "spec": { "workflows": [ { "name": "my_etl", "script": { "path": "my_etl", "runtime": { "command": "WORKFLOW" } }, "trigger": { "type": "Scheduler", "cron": "00 00 00 * * ?", "startTime": "1970-01-01 00:00:00", "endTime": "9999-01-01 00:00:00", "timezone": "Asia/Shanghai" } } ] } } FILE:assets/templates/05-cycle-workflow/my_etl/transform/transform.spec.json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "transform", "id": "transform", "recurrence": "Normal", "timeout": 4, "timeoutUnit": "HOURS", "instanceMode": "T+1", "rerunMode": "Allowed", "rerunTimes": 0, "rerunInterval": 180000, "script": { "path": "transform", "language": "odps-sql", "runtime": { "command": "ODPS_SQL" }, "content": "", "parameters": [ { "name": "bizdate", "scope": "NodeParameter", "type": "System", "value": "$yyyymmdd", "artifactType": "Variable" } ] }, "trigger": { "type": "Scheduler", "cron": "00 00 00 * * ?", "startTime": "1970-01-01 00:00:00", "endTime": "9999-01-01 00:00:00", "timezone": "Asia/Shanghai" }, "datasource": { "name": "${spec.datasource.name}", "type": "odps" }, "runtimeResource": { "resourceGroup": "${spec.runtimeResource.resourceGroup}" }, "outputs": { "nodeOutputs": [ { "data": "${projectIdentifier}.transform", "artifactType": "NodeOutput" } ] }, "inputs": { "nodeOutputs": [ { "data": "${projectIdentifier}.extract", "artifactType": "NodeOutput" } ] } } ], "dependencies": [ { "nodeId": "transform", "depends": [ { "type": "Normal", "output": "${projectIdentifier}.extract" } ] } ] } } FILE:assets/templates/05-cycle-workflow/my_etl/transform/transform.sql -- Transform: Data cleansing and transformation INSERT OVERWRITE TABLE dwd_order PARTITION (dt='${bizdate}') SELECT order_id, user_id, COALESCE(amount, 0) AS amount, CASE WHEN status = 1 THEN 'completed' ELSE 'pending' END AS status, order_time FROM ods_raw_order WHERE dt = '${bizdate}' AND order_id IS NOT NULL; FILE:assets/templates/06-manual-workflow/README.md # Example 06: Manual Workflow Manually triggered workflow with two steps. ## File Structure ``` manual_task/ ├── manual_task.spec.json ├── dataworks.properties ├── step1/ │ ├── step1.spec.json │ ├── step1.sh │ └── dataworks.properties └── step2/ ├── step2.spec.json ├── step2.py └── dataworks.properties ``` FILE:assets/templates/06-manual-workflow/manual_task/manual_task.spec.json { "version": "2.0.0", "kind": "ManualWorkflow", "spec": { "workflows": [ { "name": "manual_task", "script": { "path": "manual_task", "runtime": { "command": "WORKFLOW" } } } ] } } FILE:assets/templates/06-manual-workflow/manual_task/step1/step1.sh #!/bin/bash echo "Manual step 1: Preparing environment" echo "Done." FILE:assets/templates/06-manual-workflow/manual_task/step1/step1.spec.json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "step1", "id": "step1", "recurrence": "Normal", "timeout": 1, "timeoutUnit": "HOURS", "instanceMode": "T+1", "rerunMode": "Allowed", "rerunTimes": 0, "rerunInterval": 180000, "script": { "path": "step1", "language": "shell", "runtime": { "command": "DIDE_SHELL" }, "content": "" }, "trigger": { "type": "Manual" }, "runtimeResource": { "resourceGroup": "${spec.runtimeResource.resourceGroup}" }, "outputs": { "nodeOutputs": [ { "data": "${projectIdentifier}.step1", "artifactType": "NodeOutput" } ] }, "inputs": { "nodeOutputs": [ { "data": "${projectIdentifier}_root", "artifactType": "NodeOutput" } ] } } ], "dependencies": [ { "nodeId": "step1", "depends": [ { "type": "Normal", "output": "${projectIdentifier}_root" } ] } ] } } FILE:assets/templates/06-manual-workflow/manual_task/step2/step2.py # Manual step 2: Data processing import sys print("Manual step 2: Processing data") print(f"Python version: {sys.version}") print("Processing completed successfully.") FILE:assets/templates/06-manual-workflow/manual_task/step2/step2.spec.json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "step2", "id": "step2", "recurrence": "Normal", "timeout": 2, "timeoutUnit": "HOURS", "instanceMode": "T+1", "rerunMode": "Allowed", "rerunTimes": 0, "rerunInterval": 180000, "script": { "path": "step2", "language": "python", "runtime": { "command": "PYTHON" }, "content": "" }, "trigger": { "type": "Manual" }, "runtimeResource": { "resourceGroup": "${spec.runtimeResource.resourceGroup}" }, "outputs": { "nodeOutputs": [ { "data": "${projectIdentifier}.step2", "artifactType": "NodeOutput" } ] }, "inputs": { "nodeOutputs": [ { "data": "${projectIdentifier}.step1", "artifactType": "NodeOutput" } ] } } ], "dependencies": [ { "nodeId": "step2", "depends": [ { "type": "Normal", "output": "${projectIdentifier}.step1" } ] } ] } } FILE:assets/templates/07-parallel-workflow/README.md # Example 07: Parallel Workflow (Fan-out + Fan-in) Parallel ETL workflow with 4 child nodes, demonstrating fan-out and fan-in dependency patterns: ``` prepare_data (Shell) ├──→ process_orders (SQL) ──┐ └──→ process_users (SQL) ──┴──→ merge_report (SQL) ``` Dependency scenarios covered: - Root node dependency (prepare_data <- project_root) - Fan-out: multiple nodes depend on the same upstream (process_orders, process_users <- prepare_data) - Fan-in: one node depends on multiple upstreams (merge_report <- process_orders + process_users) ## File Structure ``` parallel_etl/ ├── parallel_etl.spec.json # Workflow definition ├── dataworks.properties ├── prepare_data/ │ ├── prepare_data.spec.json │ ├── prepare_data.sh │ └── dataworks.properties ├── process_orders/ │ ├── process_orders.spec.json │ ├── process_orders.sql │ └── dataworks.properties ├── process_users/ │ ├── process_users.spec.json │ ├── process_users.sql │ └── dataworks.properties └── merge_report/ ├── merge_report.spec.json ├── merge_report.sql └── dataworks.properties ``` ## Dependency Configuration Key Points When creating nodes inside a workflow using CreateNode + ContainerId, dependencies are configured via `spec.dependencies` -- there is no need to dual-write `inputs.nodeOutputs`. Note: `spec.dependencies[*].nodeId` MUST exactly match the corresponding node's `id`, otherwise the dependency information will not be recognized. Multi-upstream dependency (fan-in) example (merge_report depends on both process_orders and process_users): ```json "inputs": { "nodeOutputs": [ {"data": "${projectIdentifier}.process_orders", "artifactType": "NodeOutput"}, {"data": "${projectIdentifier}.process_users", "artifactType": "NodeOutput"} ] } ``` ## Deployment Order 1. Create workflow -> obtain WorkflowId 2. Create prepare_data node (ContainerId=WorkflowId) 3. Create process_orders node (ContainerId=WorkflowId) 4. Create process_users node (ContainerId=WorkflowId) 5. Create merge_report node (ContainerId=WorkflowId) 6. Publish and go live ## API Verification Status This template has been verified through CreateNode + ContainerId API testing (cn-beijing, 2026-03-28). All dependency relationships were saved correctly. FILE:assets/templates/07-parallel-workflow/_deploy.md # Deployment Command Sequence ```bash PROJECT_ID=123456 # 1. Create workflow # Note: Workflow spec must include script.runtime.command="WORKFLOW" aliyun dataworks-public CreateWorkflowDefinition \ --ProjectId $PROJECT_ID \ --Spec "$(cat /tmp/wf.json)" # -> Record the returned WorkflowId WF_ID="" # 2. Create prepare_data node (root node, depends on project_root) # Note: Dependencies are configured via spec.dependencies; ensure dependencies[*].nodeId exactly matches the node id aliyun dataworks-public CreateNode \ --ProjectId $PROJECT_ID \ --Scene DATAWORKS_PROJECT \ --ContainerId $WF_ID \ --Spec "$(cat /tmp/n1.json)" # 3. Create process_orders node (fan-out, depends on prepare_data) aliyun dataworks-public CreateNode \ --ProjectId $PROJECT_ID \ --Scene DATAWORKS_PROJECT \ --ContainerId $WF_ID \ --Spec "$(cat /tmp/n2.json)" # 4. Create process_users node (fan-out, depends on prepare_data) aliyun dataworks-public CreateNode \ --ProjectId $PROJECT_ID \ --Scene DATAWORKS_PROJECT \ --ContainerId $WF_ID \ --Spec "$(cat /tmp/n3.json)" # 5. Create merge_report node (fan-in, depends on process_orders + process_users) # Note: Multiple upstream dependencies are listed as multiple entries in the spec.dependencies depends array aliyun dataworks-public CreateNode \ --ProjectId $PROJECT_ID \ --Scene DATAWORKS_PROJECT \ --ContainerId $WF_ID \ --Spec "$(cat /tmp/n4.json)" # 6. Publish and go live aliyun dataworks-public CreatePipelineRun \ --ProjectId $PROJECT_ID \ --Type Online \ --ObjectIds "[\"$WF_ID\"]" ``` FILE:assets/templates/07-parallel-workflow/parallel_etl/merge_report/merge_report.spec.json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "merge_report", "id": "merge_report", "recurrence": "Normal", "timeout": 4, "timeoutUnit": "HOURS", "instanceMode": "T+1", "rerunMode": "Allowed", "rerunTimes": 0, "rerunInterval": 180000, "script": { "path": "merge_report", "language": "odps-sql", "runtime": { "command": "ODPS_SQL" }, "content": "", "parameters": [ { "name": "bizdate", "scope": "NodeParameter", "type": "System", "value": "$yyyymmdd", "artifactType": "Variable" } ] }, "trigger": { "type": "Scheduler", "cron": "00 30 01 * * ?", "startTime": "1970-01-01 00:00:00", "endTime": "9999-01-01 00:00:00", "timezone": "Asia/Shanghai" }, "datasource": { "name": "${spec.datasource.name}", "type": "odps" }, "runtimeResource": { "resourceGroup": "${spec.runtimeResource.resourceGroup}" }, "outputs": { "nodeOutputs": [ { "data": "${projectIdentifier}.merge_report", "artifactType": "NodeOutput" } ] }, "inputs": { "nodeOutputs": [ { "data": "${projectIdentifier}.process_orders", "artifactType": "NodeOutput" }, { "data": "${projectIdentifier}.process_users", "artifactType": "NodeOutput" } ] } } ], "dependencies": [ { "nodeId": "merge_report", "depends": [ { "type": "Normal", "output": "${projectIdentifier}.process_orders" }, { "type": "Normal", "output": "${projectIdentifier}.process_users" } ] } ] } } FILE:assets/templates/07-parallel-workflow/parallel_etl/merge_report/merge_report.sql -- merge_report: Merge order and user data to generate report INSERT OVERWRITE TABLE ads_user_order_report PARTITION (dt='${bizdate}') SELECT u.user_id, u.user_name, u.region, COUNT(o.order_id) AS order_count, SUM(o.amount) AS total_amount FROM dwd_user u LEFT JOIN dwd_order o ON u.user_id = o.user_id AND o.dt = '${bizdate}' WHERE u.dt = '${bizdate}' GROUP BY u.user_id, u.user_name, u.region; FILE:assets/templates/07-parallel-workflow/parallel_etl/parallel_etl.spec.json { "version": "2.0.0", "kind": "CycleWorkflow", "spec": { "workflows": [ { "name": "parallel_etl", "script": { "path": "parallel_etl", "runtime": { "command": "WORKFLOW" } }, "trigger": { "type": "Scheduler", "cron": "00 30 01 * * ?", "startTime": "1970-01-01 00:00:00", "endTime": "9999-01-01 00:00:00", "timezone": "Asia/Shanghai" } } ] } } FILE:assets/templates/07-parallel-workflow/parallel_etl/prepare_data/prepare_data.sh #!/bin/bash # prepare_data: Check source data readiness echo "Checking source data for date: $bizdate" # Actual logic: Verify upstream data is available echo "Source data is ready." FILE:assets/templates/07-parallel-workflow/parallel_etl/prepare_data/prepare_data.spec.json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "prepare_data", "id": "prepare_data", "recurrence": "Normal", "timeout": 2, "timeoutUnit": "HOURS", "instanceMode": "T+1", "rerunMode": "Allowed", "rerunTimes": 3, "rerunInterval": 180000, "script": { "path": "prepare_data", "language": "shell", "runtime": { "command": "DIDE_SHELL" }, "content": "", "parameters": [ { "name": "bizdate", "scope": "NodeParameter", "type": "System", "value": "$yyyymmdd", "artifactType": "Variable" } ] }, "trigger": { "type": "Scheduler", "cron": "00 30 01 * * ?", "startTime": "1970-01-01 00:00:00", "endTime": "9999-01-01 00:00:00", "timezone": "Asia/Shanghai" }, "runtimeResource": { "resourceGroup": "${spec.runtimeResource.resourceGroup}" }, "outputs": { "nodeOutputs": [ { "data": "${projectIdentifier}.prepare_data", "artifactType": "NodeOutput" } ] }, "inputs": { "nodeOutputs": [ { "data": "${projectIdentifier}_root", "artifactType": "NodeOutput" } ] } } ], "dependencies": [ { "nodeId": "prepare_data", "depends": [ { "type": "Normal", "output": "${projectIdentifier}_root" } ] } ] } } FILE:assets/templates/07-parallel-workflow/parallel_etl/process_orders/process_orders.spec.json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "process_orders", "id": "process_orders", "recurrence": "Normal", "timeout": 4, "timeoutUnit": "HOURS", "instanceMode": "T+1", "rerunMode": "Allowed", "rerunTimes": 0, "rerunInterval": 180000, "script": { "path": "process_orders", "language": "odps-sql", "runtime": { "command": "ODPS_SQL" }, "content": "", "parameters": [ { "name": "bizdate", "scope": "NodeParameter", "type": "System", "value": "$yyyymmdd", "artifactType": "Variable" } ] }, "trigger": { "type": "Scheduler", "cron": "00 30 01 * * ?", "startTime": "1970-01-01 00:00:00", "endTime": "9999-01-01 00:00:00", "timezone": "Asia/Shanghai" }, "datasource": { "name": "${spec.datasource.name}", "type": "odps" }, "runtimeResource": { "resourceGroup": "${spec.runtimeResource.resourceGroup}" }, "outputs": { "nodeOutputs": [ { "data": "${projectIdentifier}.process_orders", "artifactType": "NodeOutput" } ] }, "inputs": { "nodeOutputs": [ { "data": "${projectIdentifier}.prepare_data", "artifactType": "NodeOutput" } ] } } ], "dependencies": [ { "nodeId": "process_orders", "depends": [ { "type": "Normal", "output": "${projectIdentifier}.prepare_data" } ] } ] } } FILE:assets/templates/07-parallel-workflow/parallel_etl/process_orders/process_orders.sql -- process_orders: Order data cleansing INSERT OVERWRITE TABLE dwd_order PARTITION (dt='${bizdate}') SELECT order_id, user_id, COALESCE(amount, 0) AS amount, order_time FROM ods_raw_order WHERE dt = '${bizdate}' AND order_id IS NOT NULL; FILE:assets/templates/07-parallel-workflow/parallel_etl/process_users/process_users.spec.json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "process_users", "id": "process_users", "recurrence": "Normal", "timeout": 4, "timeoutUnit": "HOURS", "instanceMode": "T+1", "rerunMode": "Allowed", "rerunTimes": 0, "rerunInterval": 180000, "script": { "path": "process_users", "language": "odps-sql", "runtime": { "command": "ODPS_SQL" }, "content": "", "parameters": [ { "name": "bizdate", "scope": "NodeParameter", "type": "System", "value": "$yyyymmdd", "artifactType": "Variable" } ] }, "trigger": { "type": "Scheduler", "cron": "00 30 01 * * ?", "startTime": "1970-01-01 00:00:00", "endTime": "9999-01-01 00:00:00", "timezone": "Asia/Shanghai" }, "datasource": { "name": "${spec.datasource.name}", "type": "odps" }, "runtimeResource": { "resourceGroup": "${spec.runtimeResource.resourceGroup}" }, "outputs": { "nodeOutputs": [ { "data": "${projectIdentifier}.process_users", "artifactType": "NodeOutput" } ] }, "inputs": { "nodeOutputs": [ { "data": "${projectIdentifier}.prepare_data", "artifactType": "NodeOutput" } ] } } ], "dependencies": [ { "nodeId": "process_users", "depends": [ { "type": "Normal", "output": "${projectIdentifier}.prepare_data" } ] } ] } } FILE:assets/templates/07-parallel-workflow/parallel_etl/process_users/process_users.sql -- process_users: User data cleansing INSERT OVERWRITE TABLE dwd_user PARTITION (dt='${bizdate}') SELECT user_id, user_name, COALESCE(region, 'unknown') AS region, register_time FROM ods_raw_user WHERE dt = '${bizdate}' AND user_id IS NOT NULL; FILE:assets/templates/README.md # DataWorks Data Development Examples | Example | Scenario | Node Type | Complexity | |---------|----------|-----------|------------| | [01-shell-node](01-shell-node/) | Simplest node | DIDE_SHELL | Beginner | | [02-odps-sql-node](02-odps-sql-node/) | SQL node with datasource | ODPS_SQL | Beginner | | [03-sql-with-dependency](03-sql-with-dependency/) | Inter-node dependency | ODPS_SQL | Intermediate | | [04-di-mysql-to-maxcompute](04-di-mysql-to-maxcompute/) | Data synchronization | DI | Intermediate | | [05-cycle-workflow](05-cycle-workflow/) | Scheduled workflow with 3 child nodes | Mixed | Advanced | | [06-manual-workflow](06-manual-workflow/) | Manual workflow | Mixed | Advanced | | [07-parallel-workflow](07-parallel-workflow/) | Parallel workflow (fan-out + fan-in dependencies) | Mixed | Advanced | Each example contains spec.json, code files, and dataworks.properties, demonstrating the local file structure in Git Mode. FILE:references/acceptance-criteria.md # Acceptance Criteria: DataWorks Data Development **Scenario**: DataWorks node and workflow development **Purpose**: Skill test acceptance criteria --- ## Correct CLI Command Patterns ### 1. Product -- Verify Product Name Exists ```bash # CORRECT: dataworks-public is the correct product name aliyun dataworks-public GetNode --help # INCORRECT: dataworks without the -public suffix aliyun dataworks GetNode --help ``` ### 2. Command -- Verify Action Exists Under the Product ```bash # CORRECT: Correct action names aliyun dataworks-public CreateNode --help aliyun dataworks-public ListNodes --help aliyun dataworks-public GetNode --help aliyun dataworks-public UpdateNode --help aliyun dataworks-public CreateWorkflowDefinition --help aliyun dataworks-public ListWorkflowDefinitions --help aliyun dataworks-public GetWorkflowDefinition --help aliyun dataworks-public CreatePipelineRun --help aliyun dataworks-public GetPipelineRun --help aliyun dataworks-public ExecPipelineRunStage --help # INCORRECT: Wrong action names aliyun dataworks-public CreateTask --help # Should be CreateNode aliyun dataworks-public ListTask --help # Should be ListNodes ``` ### 3. Parameters -- Verify Each Parameter Name Exists ```bash # CORRECT: Correct parameter names aliyun dataworks-public CreateNode \ --ProjectId 123456 \ --Scene DATAWORKS_PROJECT \ --Spec '{"version":"2.0.0",...}' aliyun dataworks-public CreateNode \ --ProjectId 123456 \ --Scene DATAWORKS_PROJECT \ --ContainerId 789012 \ --Spec '{"version":"2.0.0",...}' # INCORRECT: Wrong parameter names aliyun dataworks-public CreateNode \ --projectId 123456 # Should be --ProjectId (uppercase P) --scene DATAWORKS_PROJECT # Should be --Scene (uppercase S) ``` ### 4. user-agent Identifier -- Must Be Included in Every Command ```bash # CORRECT: Includes user-agent aliyun dataworks-public GetNode \ --ProjectId 123456 \ --Id 789012 \ --user-agent AlibabaCloud-Agent-Skills # INCORRECT: Missing user-agent aliyun dataworks-public GetNode \ --ProjectId 123456 \ --Id 789012 ``` --- ## Correct FlowSpec Patterns ### 1. Node spec.json Basic Structure ```json // CORRECT: Correct node spec structure { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [{ "name": "my_node", "script": { "path": "my_node", "language": "odps-sql", "runtime": { "command": "ODPS_SQL" } }, "trigger": { "type": "Scheduler", "cron": "00 00 00 * * ?", "startTime": "1970-01-01 00:00:00", "endTime": "9999-01-01 00:00:00", "timezone": "Asia/Shanghai" } }], "dependencies": [{ "nodeId": "my_node", "depends": [{ "type": "Normal", "output": "${projectIdentifier}_root" }] }] } } // INCORRECT: Missing required fields { "kind": "Node", // Missing version "spec": { "nodes": [{ "name": "my_node" // Missing script }] } } ``` ### 2. script.path Must Match name ```json // CORRECT: path matches name { "name": "etl_daily", "script": { "path": "etl_daily", // Matches name "runtime": { "command": "ODPS_SQL" } } } // INCORRECT: path does not match name { "name": "etl_daily", "script": { "path": "other_path", // API will return "script path not match name" "runtime": { "command": "ODPS_SQL" } } } ``` ### 3. Dependency Configuration (spec.dependencies) ```json // CORRECT: Configure dependencies in spec.dependencies, ensure nodeId exactly matches the node id { "spec": { "nodes": [{ "name": "downstream" }], "dependencies": [{ "nodeId": "downstream", "depends": [{ "type": "Normal", "output": "${projectIdentifier}.upstream" }] }] } } // INCORRECT: Using the old dual-write approach with inputs.nodeOutputs + flow.depends { "spec": { "nodes": [{ "name": "downstream", "inputs": { "nodeOutputs": [{ "data": "${projectIdentifier}.upstream" }] } }], "flow": [{ "nodeId": "downstream", "depends": [{ "type": "Normal", "output": "${projectIdentifier}.upstream" }] }] } } ``` ### 4. Workflow spec Must Include command: WORKFLOW ```json // CORRECT: Workflow spec { "version": "2.0.0", "kind": "CycleWorkflow", "spec": { "workflows": [{ "name": "my_workflow", "script": { "path": "my_workflow", "runtime": { "command": "WORKFLOW" // Must be set } }, "trigger": { "type": "Scheduler", "cron": "00 00 00 * * ?" } }] } } // INCORRECT: Missing command { "version": "2.0.0", "kind": "CycleWorkflow", "spec": { "workflows": [{ "name": "my_workflow", "script": { "path": "my_workflow" // Missing runtime.command, API will return an error } }] } } ``` ### 5. Datasource Type Matching ```json // CORRECT: ODPS_SQL uses odps datasource { "script": { "runtime": { "command": "ODPS_SQL" }, "language": "odps-sql" }, "datasource": { "name": "${spec.datasource.name}", "type": "odps" } } // CORRECT: HOLOGRES_SQL uses hologres datasource { "script": { "runtime": { "command": "HOLOGRES_SQL" }, "language": "hologres-sql" }, "datasource": { "name": "${spec.datasource.name}", "type": "hologres" } } // INCORRECT: Type mismatch { "script": { "runtime": { "command": "HOLOGRES_SQL" } }, "datasource": { "name": "my_ds", "type": "odps" // Should be hologres } } ``` --- ## Correct dataworks.properties Patterns ```properties # CORRECT: Correct properties format projectIdentifier=my_project_name spec.datasource.name=my_odps_datasource spec.runtimeResource.resourceGroup=S_res_group_xxx script.bizdate=20260101 # INCORRECT: Wrong key prefix datasource.name=my_ds # Should be spec.datasource.name resource_group=S_res_group_xxx # Should be spec.runtimeResource.resourceGroup # INCORRECT: Value contains placeholder spec.datasource.name=${datasource} # Value must not contain placeholders ``` --- ## Correct Python SDK Code Patterns ### 1. Import Patterns ```python # CORRECT from alibabacloud_dataworks_public20240518.client import Client from alibabacloud_dataworks_public20240518.models import CreateNodeRequest from alibabacloud_tea_openapi.models import Config # INCORRECT from alibabacloud_dataworks.client import Client # Wrong module name from dataworks.models import CreateNodeRequest # Wrong module name ``` ### 2. Client Initialization ```python # CORRECT: Use CredentialClient from alibabacloud_credentials.client import Client as CredentialClient credential = CredentialClient() config = Config(credential=credential) config.endpoint = 'dataworks.cn-hangzhou.aliyuncs.com' client = Client(config) # INCORRECT: Hardcoded AK/SK (security risk) config = Config( access_key_id='LTAI5tXXX', # Do not hardcode access_key_secret='8dXXXXXXX' # Do not hardcode ) ``` ### 3. API Calls ```python # CORRECT request = CreateNodeRequest( project_id=123456, scene='DATAWORKS_PROJECT', spec=spec_json ) response = client.create_node(request) node_id = response.body.id # INCORRECT: Wrong parameter names request = CreateNodeRequest( projectId=123456, # Should be project_id Scene='xxx' # Should be scene ) ``` --- ## Validation Commands Each CLI command should be verified with `--help`: ```bash # Verify product and action exist aliyun dataworks-public CreateNode --help aliyun dataworks-public UpdateNode --help aliyun dataworks-public GetNode --help aliyun dataworks-public ListNodes --help aliyun dataworks-public CreateWorkflowDefinition --help aliyun dataworks-public UpdateWorkflowDefinition --help aliyun dataworks-public GetWorkflowDefinition --help aliyun dataworks-public ListWorkflowDefinitions --help aliyun dataworks-public CreatePipelineRun --help aliyun dataworks-public GetPipelineRun --help aliyun dataworks-public ExecPipelineRunStage --help aliyun dataworks-public ListPipelineRuns --help aliyun dataworks-public ListPipelineRunItems --help aliyun dataworks-public AbolishPipelineRun --help aliyun dataworks-public GetProject --help aliyun dataworks-public ListDataSources --help aliyun dataworks-public ListResourceGroups --help ``` --- ## Critical Anti-Patterns to Avoid 1. **Do not dual-write inputs.nodeOutputs**: Dependencies only need to be configured in the spec.dependencies array, but `dependencies[*].nodeId` must exactly match the node `id` 2. **Do not hardcode AK/SK**: Use CredentialClient or environment variables 3. **Do not forget user-agent**: Every aliyun command must include `--user-agent AlibabaCloud-Agent-Skills` 4. **Do not assume UpdateNode works for all nodes**: Hologres nodes cannot be updated 5. **Do not skip validation**: Always run validate.py after each modification 6. **Do not echo AK/SK**: Never print credential information 7. **Do not execute write operations without confirmation**: Except for Create and read-only queries (Get/List), all Delete, Update, Move, Rename, Abolish, and other APIs that modify existing objects must be confirmed with the user first FILE:references/api/AbolishPipelineRun.md # AbolishPipelineRun > Latest API definition: https://api.aliyun.com/meta/v1/products/dataworks-public/versions/2024-05-18/apis/AbolishPipelineRun/api.json > If the call returns an error, you can obtain the latest parameter definitions from the URL above. ### Cancel Publishing **aliyun CLI**: ```bash aliyun dataworks-public AbolishPipelineRun \ --ProjectId {{project_id}} \ --Id {{pipeline_run_id}} \ --user-agent AlibabaCloud-Agent-Skills ``` **Python SDK**: ```python from alibabacloud_dataworks_public20240518.models import AbolishPipelineRunRequest client.abolish_pipeline_run(AbolishPipelineRunRequest( project_id={{project_id}}, id='{{pipeline_run_id}}' )) ``` FILE:references/api/CreateComponent.md # CreateComponent > Latest API definition: https://api.aliyun.com/meta/v1/products/dataworks-public/versions/2024-05-18/apis/CreateComponent/api.json > If the call returns an error, you can obtain the latest parameter definitions from the URL above. ### Idempotency Note This API does not support `ClientToken`. If the call times out or returns a network error, **do not blindly retry**. First check whether the component was created by calling `ListComponents` and searching by name. Only retry if the component does not exist. Always record the `RequestId` from the response for traceability. ### Create Component **aliyun CLI**: ```bash aliyun dataworks-public CreateComponent \ --ProjectId {{project_id}} \ --Spec "$(cat /tmp/component.json)" \ --user-agent AlibabaCloud-Agent-Skills ``` **Python SDK**: ```python from alibabacloud_dataworks_public20240518.models import CreateComponentRequest request = CreateComponentRequest( project_id={{project_id}}, spec=spec ) response = client.create_component(request) print(f"ComponentId: {response.body.id}") ``` FILE:references/api/CreateFunction.md # CreateFunction > Latest API definition: https://api.aliyun.com/meta/v1/products/dataworks-public/versions/2024-05-18/apis/CreateFunction/api.json > If the call returns an error, you can obtain the latest parameter definitions from the URL above. ### Idempotency Note This API does not support `ClientToken`. If the call times out or returns a network error, **do not blindly retry**. First check whether the function was created by calling `ListFunctions` and searching by name. Only retry if the function does not exist. Always record the `RequestId` from the response for traceability. ### Create Function **aliyun CLI**: ```bash # Build spec JSON (replace placeholders in spec.json with actual values, embed function definition content) aliyun dataworks-public CreateFunction \ --ProjectId {{project_id}} \ --Spec "$(cat /tmp/func.json)" \ --user-agent AlibabaCloud-Agent-Skills ``` **Python SDK**: ```python from alibabacloud_dataworks_public20240518.models import CreateFunctionRequest request = CreateFunctionRequest( project_id={{project_id}}, spec=spec ) response = client.create_function(request) print(f"FunctionId: {response.body.id}") ``` FILE:references/api/CreateNode.md # CreateNode > Latest API definition: https://api.aliyun.com/meta/v1/products/dataworks-public/versions/2024-05-18/apis/CreateNode/api.json > If the call returns an error, you can obtain the latest parameter definitions from the URL above. ### Idempotency Note This API does not support `ClientToken`. If the call times out or returns a network error, **do not blindly retry**. First check whether the node was created by calling `ListNodes --Name `. Only retry if the node does not exist. Always record the `RequestId` from the response for traceability. ### Create Node **Prerequisite**: Use build.py to merge the three files (spec.json + code file + properties) into the API input: ```bash python $SKILL/scripts/build.py ./my_node > /tmp/spec.json ``` **aliyun CLI**: ```bash aliyun dataworks-public CreateNode \ --ProjectId {{project_id}} \ --Scene DATAWORKS_PROJECT \ --Spec "$(cat /tmp/spec.json)" \ --user-agent AlibabaCloud-Agent-Skills ``` **Python SDK**: ```python from alibabacloud_credentials.client import Client as CredentialClient from alibabacloud_dataworks_public20240518.client import Client from alibabacloud_dataworks_public20240518.models import CreateNodeRequest from alibabacloud_tea_openapi.models import Config credential = CredentialClient() config = Config(credential=credential) config.endpoint = 'dataworks.{{region}}.aliyuncs.com' config.user_agent = 'AlibabaCloud-Agent-Skills' client = Client(config) with open('/tmp/spec.json') as f: spec = f.read() request = CreateNodeRequest( project_id={{project_id}}, scene='DATAWORKS_PROJECT', spec=spec ) response = client.create_node(request) print(f"NodeId: {response.body.id}") ``` ### Create Node Inside a Workflow Same as above, after merging with build.py, add `--ContainerId`: **aliyun CLI**: ```bash aliyun dataworks-public CreateNode \ --ProjectId {{project_id}} \ --Scene DATAWORKS_PROJECT \ --ContainerId {{workflow_id}} \ --Spec "$(cat /tmp/spec.json)" \ --user-agent AlibabaCloud-Agent-Skills ``` **Python SDK**: ```python request = CreateNodeRequest( project_id={{project_id}}, scene='DATAWORKS_PROJECT', container_id='{{workflow_id}}', spec=spec ) response = client.create_node(request) print(f"NodeId: {response.body.id}") ``` FILE:references/api/CreatePipelineRun.md # CreatePipelineRun > Latest API definition: https://api.aliyun.com/meta/v1/products/dataworks-public/versions/2024-05-18/apis/CreatePipelineRun/api.json > If the call returns an error, you can obtain the latest parameter definitions from the URL above. ### Idempotency Note This API does not support `ClientToken`. If the call times out or returns a network error, **do not blindly retry**. First check whether a pipeline run was already created by calling `ListPipelineRuns` and filtering by the target object. Only retry if no matching active pipeline run exists. Always record the `RequestId` from the response for traceability. ### Create Pipeline Run (Publish / Deploy) **aliyun CLI**: ```bash aliyun dataworks-public CreatePipelineRun \ --ProjectId {{project_id}} \ --Type Online \ --ObjectIds "[\"{{object_id}}\"]" \ --user-agent AlibabaCloud-Agent-Skills ``` **Python SDK**: ```python from alibabacloud_dataworks_public20240518.models import CreatePipelineRunRequest # type: Online (deploy) or Offline (undeploy) # object_ids: only the first entity and its child entities will be processed request = CreatePipelineRunRequest( project_id={{project_id}}, type='Online', object_ids=['{{object_id}}'] ) response = client.create_pipeline_run(request) run_id = response.body.id print(f"PipelineRunId: {run_id}") ``` FILE:references/api/CreateResource.md # CreateResource > Latest API definition: https://api.aliyun.com/meta/v1/products/dataworks-public/versions/2024-05-18/apis/CreateResource/api.json > If the call returns an error, you can obtain the latest parameter definitions from the URL above. ### Idempotency Note This API does not support `ClientToken`. If the call times out or returns a network error, **do not blindly retry**. First check whether the resource was created by calling `ListResources` and searching by name. Only retry if the resource does not exist. Always record the `RequestId` from the response for traceability. ### Create Resource **aliyun CLI**: ```bash # Build spec JSON (replace placeholders in spec.json with actual values, embed resource file content) aliyun dataworks-public CreateResource \ --ProjectId {{project_id}} \ --Spec "$(cat /tmp/res.json)" \ --user-agent AlibabaCloud-Agent-Skills ``` **Python SDK**: ```python from alibabacloud_dataworks_public20240518.models import CreateResourceRequest request = CreateResourceRequest( project_id={{project_id}}, spec=spec ) response = client.create_resource(request) print(f"ResourceId: {response.body.id}") ``` FILE:references/api/CreateWorkflowDefinition.md # CreateWorkflowDefinition > Latest API definition: https://api.aliyun.com/meta/v1/products/dataworks-public/versions/2024-05-18/apis/CreateWorkflowDefinition/api.json > If the call returns an error, you can obtain the latest parameter definitions from the URL above. ### Idempotency Note This API does not support `ClientToken`. If the call times out or returns a network error, **do not blindly retry**. First check whether the workflow was created by calling `ListWorkflowDefinitions` and searching by name. Only retry if the workflow does not exist. Always record the `RequestId` from the response for traceability. ### Create Workflow The workflow spec must include `script.runtime.command: "WORKFLOW"`, otherwise the creation will fail. The correct spec format is as follows: ```json { "version": "2.0.0", "kind": "CycleWorkflow", "spec": { "workflows": [{ "name": "my_workflow", "script": { "path": "my_workflow", "runtime": {"command": "WORKFLOW"} }, "trigger": { "type": "Scheduler", "cron": "00 00 02 * * ?", "startTime": "1970-01-01 00:00:00", "endTime": "9999-01-01 00:00:00", "timezone": "Asia/Shanghai" } }] } } ``` **Prerequisite**: Use build.py to merge the three files (a workflow directory typically only has spec.json + properties, no code file): ```bash python $SKILL/scripts/build.py ./my_workflow > /tmp/wf.json ``` **aliyun CLI**: ```bash aliyun dataworks-public CreateWorkflowDefinition \ --ProjectId {{project_id}} \ --Spec "$(cat /tmp/wf.json)" \ --user-agent AlibabaCloud-Agent-Skills ``` **Python SDK**: ```python from alibabacloud_dataworks_public20240518.models import CreateWorkflowDefinitionRequest with open('/tmp/wf.json') as f: spec = f.read() request = CreateWorkflowDefinitionRequest( project_id={{project_id}}, spec=spec ) response = client.create_workflow_definition(request) print(f"WorkflowId: {response.body.id}") ``` FILE:references/api/ExecPipelineRunStage.md # ExecPipelineRunStage > Latest API definition: https://api.aliyun.com/meta/v1/products/dataworks-public/versions/2024-05-18/apis/ExecPipelineRunStage/api.json > If the call returns an error, you can obtain the latest parameter definitions from the URL above. ### Advance Pipeline Run Stage **aliyun CLI**: ```bash aliyun dataworks-public ExecPipelineRunStage \ --ProjectId {{project_id}} \ --Id {{pipeline_run_id}} \ --Code {{stage_code}} \ --user-agent AlibabaCloud-Agent-Skills ``` **Python SDK**: ```python from alibabacloud_dataworks_public20240518.models import ExecPipelineRunStageRequest # code: stage code, obtained from Stages[].Code in the GetPipelineRun response # stages must be advanced in order; skipping stages is not allowed # triggered asynchronously; continue polling to confirm the result client.exec_pipeline_run_stage(ExecPipelineRunStageRequest( project_id={{project_id}}, id='{{pipeline_run_id}}', code='{{stage_code}}' # e.g., PROD_CHECK, PROD )) ``` FILE:references/api/GetComponent.md # GetComponent > Latest API definition: https://api.aliyun.com/meta/v1/products/dataworks-public/versions/2024-05-18/apis/GetComponent/api.json > If the call returns an error, you can obtain the latest parameter definitions from the URL above. ### Get Component Details **aliyun CLI**: ```bash aliyun dataworks-public GetComponent \ --ProjectId {{project_id}} \ --Id {{component_id}} \ --user-agent AlibabaCloud-Agent-Skills ``` **Python SDK**: ```python from alibabacloud_dataworks_public20240518.models import GetComponentRequest request = GetComponentRequest( project_id={{project_id}}, id='{{component_id}}' ) response = client.get_component(request) print(response.body.spec) ``` FILE:references/api/GetFunction.md # GetFunction > Latest API definition: https://api.aliyun.com/meta/v1/products/dataworks-public/versions/2024-05-18/apis/GetFunction/api.json > If the call returns an error, you can obtain the latest parameter definitions from the URL above. ### Get Function Details **aliyun CLI**: ```bash aliyun dataworks-public GetFunction \ --ProjectId {{project_id}} \ --Id {{function_id}} \ --user-agent AlibabaCloud-Agent-Skills ``` **Python SDK**: ```python from alibabacloud_dataworks_public20240518.models import GetFunctionRequest request = GetFunctionRequest( project_id={{project_id}}, id='{{function_id}}' ) response = client.get_function(request) print(response.body.spec) ``` FILE:references/api/GetNode.md # GetNode > Latest API definition: https://api.aliyun.com/meta/v1/products/dataworks-public/versions/2024-05-18/apis/GetNode/api.json > If the call returns an error, you can obtain the latest parameter definitions from the URL above. ### Get Node Details **aliyun CLI**: ```bash aliyun dataworks-public GetNode \ --ProjectId {{project_id}} \ --Id {{node_id}} \ --user-agent AlibabaCloud-Agent-Skills ``` **Python SDK**: ```python from alibabacloud_dataworks_public20240518.models import GetNodeRequest request = GetNodeRequest( project_id={{project_id}}, id='{{node_id}}' ) response = client.get_node(request) # response.body.spec contains the full FlowSpec JSON ``` FILE:references/api/GetPipelineRun.md # GetPipelineRun > Latest API definition: https://api.aliyun.com/meta/v1/products/dataworks-public/versions/2024-05-18/apis/GetPipelineRun/api.json > If the call returns an error, you can obtain the latest parameter definitions from the URL above. ### Query Pipeline Run Status **aliyun CLI**: ```bash aliyun dataworks-public GetPipelineRun \ --ProjectId {{project_id}} \ --Id {{pipeline_run_id}} \ --user-agent AlibabaCloud-Agent-Skills ``` **Python SDK**: ```python from alibabacloud_dataworks_public20240518.models import GetPipelineRunRequest response = client.get_pipeline_run(GetPipelineRunRequest( project_id={{project_id}}, id='{{pipeline_run_id}}' )) pipeline = response.body.pipeline.to_map() print(f"Status: {pipeline['Status']}") # Status: Init / Running / Success / Fail / Termination / Cancel for stage in pipeline.get('Stages', []): print(f" {stage['Code']}({stage['Status']}): {stage['Name']}") ``` FILE:references/api/GetProject.md # GetProject > Latest API definition: https://api.aliyun.com/meta/v1/products/dataworks-public/versions/2024-05-18/apis/GetProject/api.json > If the call returns an error, you can obtain the latest parameter definitions from the URL above. ### Get Project Information (retrieve projectIdentifier via projectId) **aliyun CLI**: ```bash # Retrieve project details by projectId (numeric); ProjectName is the projectIdentifier aliyun dataworks-public GetProject \ --Id {{project_id}} \ --user-agent AlibabaCloud-Agent-Skills ``` > **Note**: `GetProject` only accepts the numeric `--Id` parameter; reverse lookup by projectIdentifier is not supported. **Python SDK**: ```python from alibabacloud_dataworks_public20240518.models import GetProjectRequest request = GetProjectRequest( id={{project_id}} ) response = client.get_project(request) # ProjectName in the response is the projectIdentifier ``` FILE:references/api/GetResource.md # GetResource > Latest API definition: https://api.aliyun.com/meta/v1/products/dataworks-public/versions/2024-05-18/apis/GetResource/api.json > If the call returns an error, you can obtain the latest parameter definitions from the URL above. ### Get File Resource Details **aliyun CLI**: ```bash aliyun dataworks-public GetResource \ --ProjectId {{project_id}} \ --Id {{resource_id}} \ --user-agent AlibabaCloud-Agent-Skills ``` **Python SDK**: ```python from alibabacloud_dataworks_public20240518.models import GetResourceRequest request = GetResourceRequest( project_id={{project_id}}, id='{{resource_id}}' ) response = client.get_resource(request) print(response.body.spec) ``` FILE:references/api/GetWorkflowDefinition.md # GetWorkflowDefinition > Latest API definition: https://api.aliyun.com/meta/v1/products/dataworks-public/versions/2024-05-18/apis/GetWorkflowDefinition/api.json > If the call returns an error, you can obtain the latest parameter definitions from the URL above. ### Get Workflow Details **aliyun CLI**: ```bash aliyun dataworks-public GetWorkflowDefinition \ --ProjectId {{project_id}} \ --Id {{workflow_id}} \ --user-agent AlibabaCloud-Agent-Skills ``` **Python SDK**: ```python from alibabacloud_dataworks_public20240518.models import GetWorkflowDefinitionRequest request = GetWorkflowDefinitionRequest( project_id={{project_id}}, id='{{workflow_id}}' ) response = client.get_workflow_definition(request) ``` FILE:references/api/ImportWorkflowDefinition.md # ImportWorkflowDefinition > Latest API definition: https://api.aliyun.com/meta/v1/products/dataworks-public/versions/2024-05-18/apis/ImportWorkflowDefinition/api.json > If the call returns an error, you can obtain the latest parameter definitions from the URL above. ### Import Workflow (including internal child nodes) **aliyun CLI**: ```bash # spec contains the workflow definition and all child node definitions aliyun dataworks-public ImportWorkflowDefinition \ --ProjectId {{project_id}} \ --Spec "$(cat /tmp/wf_with_nodes.json)" \ --user-agent AlibabaCloud-Agent-Skills ``` **Python SDK**: ```python from alibabacloud_dataworks_public20240518.models import ImportWorkflowDefinitionRequest request = ImportWorkflowDefinitionRequest( project_id={{project_id}}, spec=spec ) response = client.import_workflow_definition(request) ``` FILE:references/api/ListComponents.md # ListComponents > Latest API definition: https://api.aliyun.com/meta/v1/products/dataworks-public/versions/2024-05-18/apis/ListComponents/api.json > If the call returns an error, you can obtain the latest parameter definitions from the URL above. ### List Components **aliyun CLI**: ```bash aliyun dataworks-public ListComponents \ --ProjectId {{project_id}} \ --PageNumber 1 \ --PageSize 100 \ --user-agent AlibabaCloud-Agent-Skills ``` **Python SDK**: ```python from alibabacloud_dataworks_public20240518.models import ListComponentsRequest request = ListComponentsRequest( project_id={{project_id}}, page_number=1, page_size=100 ) response = client.list_components(request) ``` FILE:references/api/ListComputeResources.md # ListComputeResources > Latest API definition: https://api.aliyun.com/meta/v1/products/dataworks-public/versions/2024-05-18/apis/ListComputeResources/api.json Query the list of compute resources bound to the project. Use this API to discover compute engine bindings (EMR Serverless Spark, Hologres, StarRocks, etc.) that may not appear in `ListDataSources`. **aliyun CLI**: ```bash aliyun dataworks-public ListComputeResources \ --ProjectId {{project_id}} \ --user-agent AlibabaCloud-Agent-Skills ``` **Key parameters**: - `--ProjectId` (Required) — DataWorks workspace ID - `--EnvType` (Optional) — `Dev` or `Prod` - `--Types.1`, `--Types.2`, ... (Optional) — Filter by compute resource type **Response fields of interest**: - `ComputeResourceList[].Name` — Compute resource name (can be used as `datasource.name`) - `ComputeResourceList[].Type` — Compute engine type (e.g., `EMR_Serverless`, `Hologres`, `StarRocks`) - `ComputeResourceList[].EnvType` — Environment type (`Dev` / `Prod`) FILE:references/api/ListDataSources.md # ListDataSources > Latest API definition: https://api.aliyun.com/meta/v1/products/dataworks-public/versions/2024-05-18/apis/ListDataSources/api.json Query the list of data sources in the project. **aliyun CLI**: ```bash aliyun dataworks-public ListDataSources \ --ProjectId {{project_id}} \ --user-agent AlibabaCloud-Agent-Skills ``` **Python SDK**: ```python from alibabacloud_dataworks_public20240518.models import ListDataSourcesRequest response = client.list_data_sources(ListDataSourcesRequest( project_id={{project_id}} )) # The response structure depends on the actual SDK version; use .to_map() to inspect ``` FILE:references/api/ListFunctions.md # ListFunctions > Latest API definition: https://api.aliyun.com/meta/v1/products/dataworks-public/versions/2024-05-18/apis/ListFunctions/api.json > If the call returns an error, you can obtain the latest parameter definitions from the URL above. ### List Functions **aliyun CLI**: ```bash aliyun dataworks-public ListFunctions \ --ProjectId {{project_id}} \ --PageNumber 1 \ --PageSize 100 \ --user-agent AlibabaCloud-Agent-Skills ``` ## Node Dependency Configuration FILE:references/api/ListNodeDependencies.md # ListNodeDependencies > Latest API definition: https://api.aliyun.com/meta/v1/products/dataworks-public/versions/2024-05-18/apis/ListNodeDependencies/api.json > If the call returns an error, you can obtain the latest parameter definitions from the URL above. ### List Node Dependencies **aliyun CLI**: ```bash aliyun dataworks-public ListNodeDependencies \ --ProjectId {{project_id}} \ --Id {{node_id}} \ --PageNumber 1 \ --PageSize 100 \ --user-agent AlibabaCloud-Agent-Skills ``` **Python SDK**: ```python from alibabacloud_dataworks_public20240518.models import ListNodeDependenciesRequest request = ListNodeDependenciesRequest( project_id={{project_id}}, id='{{node_id}}', page_number=1, page_size=100 ) response = client.list_node_dependencies(request) ``` FILE:references/api/ListNodes.md # ListNodes > Latest API definition: https://api.aliyun.com/meta/v1/products/dataworks-public/versions/2024-05-18/apis/ListNodes/api.json > If the call returns an error, you can obtain the latest parameter definitions from the URL above. ### List Nodes **aliyun CLI**: ```bash aliyun dataworks-public ListNodes \ --ProjectId {{project_id}} \ --Scene DATAWORKS_PROJECT \ --PageNumber 1 \ --PageSize 100 \ --user-agent AlibabaCloud-Agent-Skills ``` **Python SDK**: ```python from alibabacloud_dataworks_public20240518.models import ListNodesRequest request = ListNodesRequest( project_id={{project_id}}, scene='DATAWORKS_PROJECT', page_number=1, page_size=100 ) response = client.list_nodes(request) for node in response.body.paging_info.nodes: print(f"{node.id}: {node.name}") ``` FILE:references/api/ListPipelineRunItems.md # ListPipelineRunItems > Latest API definition: https://api.aliyun.com/meta/v1/products/dataworks-public/versions/2024-05-18/apis/ListPipelineRunItems/api.json > If the call returns an error, you can obtain the latest parameter definitions from the URL above. ### List Pipeline Run Items **aliyun CLI**: ```bash aliyun dataworks-public ListPipelineRunItems \ --ProjectId {{project_id}} \ --PipelineRunId {{pipeline_run_id}} \ --PageNumber 1 \ --PageSize 50 \ --user-agent AlibabaCloud-Agent-Skills ``` **Python SDK**: ```python from alibabacloud_dataworks_public20240518.models import ListPipelineRunItemsRequest response = client.list_pipeline_run_items(ListPipelineRunItemsRequest( project_id={{project_id}}, pipeline_run_id='{{pipeline_run_id}}', page_number=1, page_size=50 )) for item in response.body.paging_info.pipeline_run_items: m = item.to_map() print(f"{m['Name']}: {m.get('Status', 'N/A')}") ``` FILE:references/api/ListPipelineRuns.md # ListPipelineRuns > Latest API definition: https://api.aliyun.com/meta/v1/products/dataworks-public/versions/2024-05-18/apis/ListPipelineRuns/api.json > If the call returns an error, you can obtain the latest parameter definitions from the URL above. ### List Pipeline Run History **aliyun CLI**: ```bash aliyun dataworks-public ListPipelineRuns \ --ProjectId {{project_id}} \ --PageNumber 1 \ --PageSize 20 \ --user-agent AlibabaCloud-Agent-Skills ``` **Python SDK**: ```python from alibabacloud_dataworks_public20240518.models import ListPipelineRunsRequest response = client.list_pipeline_runs(ListPipelineRunsRequest( project_id={{project_id}}, page_number=1, page_size=20 )) for run in response.body.paging_info.pipeline_runs: m = run.to_map() print(f"{m['Id']} [{m['Status']}]") ``` FILE:references/api/ListResourceGroups.md # ListResourceGroups > Latest API definition: https://api.aliyun.com/meta/v1/products/dataworks-public/versions/2024-05-18/apis/ListResourceGroups/api.json Query the list of resource groups in the project. **aliyun CLI**: ```bash aliyun dataworks-public ListResourceGroups \ --ProjectId {{project_id}} \ --user-agent AlibabaCloud-Agent-Skills ``` **Python SDK**: ```python from alibabacloud_dataworks_public20240518.models import ListResourceGroupsRequest response = client.list_resource_groups(ListResourceGroupsRequest( project_id={{project_id}} )) # The response structure depends on the actual SDK version; use .to_map() to inspect ``` FILE:references/api/ListResources.md # ListResources > Latest API definition: https://api.aliyun.com/meta/v1/products/dataworks-public/versions/2024-05-18/apis/ListResources/api.json > If the call returns an error, you can obtain the latest parameter definitions from the URL above. ### List Resources **aliyun CLI**: ```bash aliyun dataworks-public ListResources \ --ProjectId {{project_id}} \ --PageNumber 1 \ --PageSize 100 \ --user-agent AlibabaCloud-Agent-Skills ``` FILE:references/api/ListWorkflowDefinitions.md # ListWorkflowDefinitions > Latest API definition: https://api.aliyun.com/meta/v1/products/dataworks-public/versions/2024-05-18/apis/ListWorkflowDefinitions/api.json > If the call returns an error, you can obtain the latest parameter definitions from the URL above. ### List Workflows **aliyun CLI**: ```bash aliyun dataworks-public ListWorkflowDefinitions \ --ProjectId {{project_id}} \ --Type CycleWorkflow \ --PageNumber 1 \ --PageSize 100 \ --user-agent AlibabaCloud-Agent-Skills ``` **Python SDK**: ```python from alibabacloud_dataworks_public20240518.models import ListWorkflowDefinitionsRequest request = ListWorkflowDefinitionsRequest( project_id={{project_id}}, type='CycleWorkflow', page_number=1, page_size=100 ) response = client.list_workflow_definitions(request) ``` ## File Resource Operations FILE:references/api/MoveFunction.md # MoveFunction > Latest API definition: https://api.aliyun.com/meta/v1/products/dataworks-public/versions/2024-05-18/apis/MoveFunction/api.json > If the call returns an error, you can obtain the latest parameter definitions from the URL above. ### Move Function to Target Path **aliyun CLI**: ```bash aliyun dataworks-public MoveFunction \ --ProjectId {{project_id}} \ --Id {{function_id}} \ --Path {{target_path}} \ --user-agent AlibabaCloud-Agent-Skills ``` **Python SDK**: ```python from alibabacloud_dataworks_public20240518.models import MoveFunctionRequest request = MoveFunctionRequest( project_id={{project_id}}, id='{{function_id}}', path='{{target_path}}' ) client.move_function(request) ``` FILE:references/api/MoveNode.md # MoveNode > Latest API definition: https://api.aliyun.com/meta/v1/products/dataworks-public/versions/2024-05-18/apis/MoveNode/api.json > If the call returns an error, you can obtain the latest parameter definitions from the URL above. ### Move Node Path **aliyun CLI**: ```bash aliyun dataworks-public MoveNode \ --ProjectId {{project_id}} \ --Id {{node_id}} \ --Path {{target_path}} \ --user-agent AlibabaCloud-Agent-Skills ``` **Python SDK**: ```python from alibabacloud_dataworks_public20240518.models import MoveNodeRequest request = MoveNodeRequest( project_id={{project_id}}, id='{{node_id}}', path='{{target_path}}' ) client.move_node(request) ``` FILE:references/api/MoveResource.md # MoveResource > Latest API definition: https://api.aliyun.com/meta/v1/products/dataworks-public/versions/2024-05-18/apis/MoveResource/api.json > If the call returns an error, you can obtain the latest parameter definitions from the URL above. ### Move File Resource to Target Directory **aliyun CLI**: ```bash aliyun dataworks-public MoveResource \ --ProjectId {{project_id}} \ --Id {{resource_id}} \ --Path {{target_path}} \ --user-agent AlibabaCloud-Agent-Skills ``` **Python SDK**: ```python from alibabacloud_dataworks_public20240518.models import MoveResourceRequest request = MoveResourceRequest( project_id={{project_id}}, id='{{resource_id}}', path='{{target_path}}' ) client.move_resource(request) ``` FILE:references/api/MoveWorkflowDefinition.md # MoveWorkflowDefinition > Latest API definition: https://api.aliyun.com/meta/v1/products/dataworks-public/versions/2024-05-18/apis/MoveWorkflowDefinition/api.json > If the call returns an error, you can obtain the latest parameter definitions from the URL above. ### Move Workflow to Target Path **aliyun CLI**: ```bash aliyun dataworks-public MoveWorkflowDefinition \ --ProjectId {{project_id}} \ --Id {{workflow_id}} \ --Path {{target_path}} \ --user-agent AlibabaCloud-Agent-Skills ``` **Python SDK**: ```python from alibabacloud_dataworks_public20240518.models import MoveWorkflowDefinitionRequest request = MoveWorkflowDefinitionRequest( project_id={{project_id}}, id='{{workflow_id}}', path='{{target_path}}' ) client.move_workflow_definition(request) ``` FILE:references/api/RenameFunction.md # RenameFunction > Latest API definition: https://api.aliyun.com/meta/v1/products/dataworks-public/versions/2024-05-18/apis/RenameFunction/api.json > If the call returns an error, you can obtain the latest parameter definitions from the URL above. ### Rename Function **aliyun CLI**: ```bash aliyun dataworks-public RenameFunction \ --ProjectId {{project_id}} \ --Id {{function_id}} \ --Name {{new_name}} \ --user-agent AlibabaCloud-Agent-Skills ``` **Python SDK**: ```python from alibabacloud_dataworks_public20240518.models import RenameFunctionRequest request = RenameFunctionRequest( project_id={{project_id}}, id='{{function_id}}', name='{{new_name}}' ) client.rename_function(request) ``` FILE:references/api/RenameNode.md # RenameNode > Latest API definition: https://api.aliyun.com/meta/v1/products/dataworks-public/versions/2024-05-18/apis/RenameNode/api.json > If the call returns an error, you can obtain the latest parameter definitions from the URL above. ### Rename Node **aliyun CLI**: ```bash aliyun dataworks-public RenameNode \ --ProjectId {{project_id}} \ --Id {{node_id}} \ --Name {{new_name}} \ --user-agent AlibabaCloud-Agent-Skills ``` **Python SDK**: ```python from alibabacloud_dataworks_public20240518.models import RenameNodeRequest request = RenameNodeRequest( project_id={{project_id}}, id='{{node_id}}', name='{{new_name}}' ) client.rename_node(request) ``` FILE:references/api/RenameResource.md # RenameResource > Latest API definition: https://api.aliyun.com/meta/v1/products/dataworks-public/versions/2024-05-18/apis/RenameResource/api.json > If the call returns an error, you can obtain the latest parameter definitions from the URL above. ### Rename File Resource **aliyun CLI**: ```bash aliyun dataworks-public RenameResource \ --ProjectId {{project_id}} \ --Id {{resource_id}} \ --Name {{new_name}} \ --user-agent AlibabaCloud-Agent-Skills ``` **Python SDK**: ```python from alibabacloud_dataworks_public20240518.models import RenameResourceRequest request = RenameResourceRequest( project_id={{project_id}}, id='{{resource_id}}', name='{{new_name}}' ) client.rename_resource(request) ``` FILE:references/api/RenameWorkflowDefinition.md # RenameWorkflowDefinition > Latest API definition: https://api.aliyun.com/meta/v1/products/dataworks-public/versions/2024-05-18/apis/RenameWorkflowDefinition/api.json > If the call returns an error, you can obtain the latest parameter definitions from the URL above. ### Rename Workflow **aliyun CLI**: ```bash aliyun dataworks-public RenameWorkflowDefinition \ --ProjectId {{project_id}} \ --Id {{workflow_id}} \ --Name {{new_name}} \ --user-agent AlibabaCloud-Agent-Skills ``` **Python SDK**: ```python from alibabacloud_dataworks_public20240518.models import RenameWorkflowDefinitionRequest request = RenameWorkflowDefinitionRequest( project_id={{project_id}}, id='{{workflow_id}}', name='{{new_name}}' ) client.rename_workflow_definition(request) ``` FILE:references/api/UpdateComponent.md # UpdateComponent > Latest API definition: https://api.aliyun.com/meta/v1/products/dataworks-public/versions/2024-05-18/apis/UpdateComponent/api.json > If the call returns an error, you can obtain the latest parameter definitions from the URL above. ### Update Component **aliyun CLI**: ```bash aliyun dataworks-public UpdateComponent \ --ProjectId {{project_id}} \ --Id {{component_id}} \ --Spec "$(cat /tmp/component.json)" \ --user-agent AlibabaCloud-Agent-Skills ``` **Python SDK**: ```python from alibabacloud_dataworks_public20240518.models import UpdateComponentRequest request = UpdateComponentRequest( project_id={{project_id}}, id='{{component_id}}', spec=spec ) client.update_component(request) ``` FILE:references/api/UpdateFunction.md # UpdateFunction > Latest API definition: https://api.aliyun.com/meta/v1/products/dataworks-public/versions/2024-05-18/apis/UpdateFunction/api.json > If the call returns an error, you can obtain the latest parameter definitions from the URL above. ### Update UDF Function Information (Incremental Update) **aliyun CLI**: ```bash aliyun dataworks-public UpdateFunction \ --ProjectId {{project_id}} \ --Id {{function_id}} \ --Spec "$(cat /tmp/func.json)" \ --user-agent AlibabaCloud-Agent-Skills ``` **Python SDK**: ```python from alibabacloud_dataworks_public20240518.models import UpdateFunctionRequest request = UpdateFunctionRequest( project_id={{project_id}}, id='{{function_id}}', spec=spec ) client.update_function(request) ``` FILE:references/api/UpdateNode.md # UpdateNode > Latest API definition: https://api.aliyun.com/meta/v1/products/dataworks-public/versions/2024-05-18/apis/UpdateNode/api.json > If the call returns an error, you can obtain the latest parameter definitions from the URL above. ### Update Node **Important**: Use incremental updates -- only send `id` + the fields to modify. Do not send unchanged fields like `datasource` or `runtimeResource` (the server may have corrected their values, and sending them back can cause errors). **aliyun CLI**: ```bash aliyun dataworks-public UpdateNode \ --ProjectId {{project_id}} \ --Id {{node_id}} \ --Spec "$(cat /tmp/update_spec.json)" \ --user-agent AlibabaCloud-Agent-Skills ``` **Python SDK**: ```python from alibabacloud_dataworks_public20240518.models import UpdateNodeRequest import json # Incremental update: only send id and the fields to modify update_spec = { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [{ "id": "{{node_id}}", "script": { "content": "new code content" } }] } } request = UpdateNodeRequest( project_id={{project_id}}, id='{{node_id}}', spec=json.dumps(update_spec, ensure_ascii=False) ) response = client.update_node(request) ``` FILE:references/api/UpdateResource.md # UpdateResource > Latest API definition: https://api.aliyun.com/meta/v1/products/dataworks-public/versions/2024-05-18/apis/UpdateResource/api.json > If the call returns an error, you can obtain the latest parameter definitions from the URL above. ### Update File Resource Information (Incremental Update) **aliyun CLI**: ```bash aliyun dataworks-public UpdateResource \ --ProjectId {{project_id}} \ --Id {{resource_id}} \ --Spec "$(cat /tmp/res.json)" \ --user-agent AlibabaCloud-Agent-Skills ``` **Python SDK**: ```python from alibabacloud_dataworks_public20240518.models import UpdateResourceRequest request = UpdateResourceRequest( project_id={{project_id}}, id='{{resource_id}}', spec=spec ) client.update_resource(request) ``` FILE:references/api/UpdateWorkflowDefinition.md # UpdateWorkflowDefinition > Latest API definition: https://api.aliyun.com/meta/v1/products/dataworks-public/versions/2024-05-18/apis/UpdateWorkflowDefinition/api.json > If the call returns an error, you can obtain the latest parameter definitions from the URL above. ### Update Workflow **aliyun CLI**: ```bash aliyun dataworks-public UpdateWorkflowDefinition \ --ProjectId {{project_id}} \ --Id {{workflow_id}} \ --Spec "$(cat /tmp/wf.json)" \ --user-agent AlibabaCloud-Agent-Skills ``` **Python SDK**: ```python from alibabacloud_dataworks_public20240518.models import UpdateWorkflowDefinitionRequest request = UpdateWorkflowDefinitionRequest( project_id={{project_id}}, id='{{workflow_id}}', spec=spec ) client.update_workflow_definition(request) ``` FILE:references/api-recipes.md # DataWorks Data Development API Call Templates All APIs are based on the DataWorks OpenAPI **2024-05-18** version. Each operation provides both aliyun CLI and Python SDK methods. ## Node Operations ### Create Node **aliyun CLI**: ```bash $PYTHON $SKILL/scripts/build.py ./my_node > /tmp/spec.json aliyun dataworks-public CreateNode \ --ProjectId {{project_id}} \ --Scene DATAWORKS_PROJECT \ --Spec "$(cat /tmp/spec.json)" \ --user-agent AlibabaCloud-Agent-Skills ``` **Python SDK**: ```python from alibabacloud_credentials.client import Client as CredentialClient from alibabacloud_dataworks_public20240518.client import Client from alibabacloud_dataworks_public20240518.models import CreateNodeRequest from alibabacloud_tea_openapi.models import Config credential = CredentialClient() config = Config(credential=credential) config.endpoint = 'dataworks.{{region}}.aliyuncs.com' config.user_agent = 'AlibabaCloud-Agent-Skills' client = Client(config) with open('/tmp/spec.json') as f: spec = f.read() request = CreateNodeRequest( project_id={{project_id}}, scene='DATAWORKS_PROJECT', spec=spec ) response = client.create_node(request) print(f"NodeId: {response.body.id}") ``` ### Create Node Within a Workflow **aliyun CLI**: ```bash $PYTHON $SKILL/scripts/build.py ./my_wf/step1 > /tmp/spec.json aliyun dataworks-public CreateNode \ --ProjectId {{project_id}} \ --Scene DATAWORKS_PROJECT \ --ContainerId {{workflow_id}} \ --Spec "$(cat /tmp/spec.json)" \ --user-agent AlibabaCloud-Agent-Skills ``` **Python SDK**: ```python request = CreateNodeRequest( project_id=383839, scene='DATAWORKS_PROJECT', container_id='', # Create the node inside a workflow spec=spec ) response = client.create_node(request) print(f"NodeId: {response.body.id}") ``` ### Update Node **aliyun CLI**: ```bash $PYTHON $SKILL/scripts/build.py ./my_node > /tmp/spec.json aliyun dataworks-public UpdateNode \ --ProjectId {{project_id}} \ --Id {{node_id}} \ --Spec "$(cat /tmp/spec.json)" \ --user-agent AlibabaCloud-Agent-Skills ``` **Python SDK**: ```python from alibabacloud_dataworks_public20240518.models import UpdateNodeRequest request = UpdateNodeRequest( project_id={{project_id}}, id='{{node_id}}', spec=spec ) response = client.update_node(request) ``` ### Get Node Details **aliyun CLI**: ```bash aliyun dataworks-public GetNode \ --ProjectId {{project_id}} \ --Id {{node_id}} \ --user-agent AlibabaCloud-Agent-Skills ``` **Python SDK**: ```python from alibabacloud_dataworks_public20240518.models import GetNodeRequest request = GetNodeRequest( project_id={{project_id}}, id='{{node_id}}' ) response = client.get_node(request) # response.body.spec contains the full FlowSpec JSON ``` ### List Nodes **aliyun CLI**: ```bash aliyun dataworks-public ListNodes \ --ProjectId {{project_id}} \ --Scene DATAWORKS_PROJECT \ --PageNumber 1 \ --PageSize 100 \ --user-agent AlibabaCloud-Agent-Skills ``` **Python SDK**: ```python from alibabacloud_dataworks_public20240518.models import ListNodesRequest request = ListNodesRequest( project_id={{project_id}}, scene='DATAWORKS_PROJECT', page_number=1, page_size=100 ) response = client.list_nodes(request) for node in response.body.paging_info.nodes: print(f"{node.id}: {node.name}") ``` ## Workflow Operations ### Create Workflow The workflow spec must include `script.runtime.command: "WORKFLOW"`, otherwise creation will fail. The correct spec format is as follows: ```json { "version": "2.0.0", "kind": "CycleWorkflow", "spec": { "workflows": [{ "name": "my_workflow", "script": { "path": "my_workflow", "runtime": {"command": "WORKFLOW"} }, "trigger": { "type": "Scheduler", "cron": "00 00 02 * * ?", "startTime": "1970-01-01 00:00:00", "endTime": "9999-01-01 00:00:00", "timezone": "Asia/Shanghai" } }] } } ``` **aliyun CLI**: ```bash $PYTHON $SKILL/scripts/build.py ./my_wf > /tmp/wf.json aliyun dataworks-public CreateWorkflowDefinition \ --ProjectId {{project_id}} \ --Spec "$(cat /tmp/wf.json)" \ --user-agent AlibabaCloud-Agent-Skills ``` **Python SDK**: ```python from alibabacloud_dataworks_public20240518.models import CreateWorkflowDefinitionRequest with open('/tmp/wf.json') as f: spec = f.read() request = CreateWorkflowDefinitionRequest( project_id={{project_id}}, spec=spec ) response = client.create_workflow_definition(request) print(f"WorkflowId: {response.body.id}") ``` ### Update Workflow **aliyun CLI**: ```bash aliyun dataworks-public UpdateWorkflowDefinition \ --ProjectId {{project_id}} \ --Id {{workflow_id}} \ --Spec "$(cat /tmp/wf.json)" \ --user-agent AlibabaCloud-Agent-Skills ``` **Python SDK**: ```python from alibabacloud_dataworks_public20240518.models import UpdateWorkflowDefinitionRequest request = UpdateWorkflowDefinitionRequest( project_id={{project_id}}, id='{{workflow_id}}', spec=spec ) client.update_workflow_definition(request) ``` ### Get Workflow Details **aliyun CLI**: ```bash aliyun dataworks-public GetWorkflowDefinition \ --ProjectId {{project_id}} \ --Id {{workflow_id}} \ --user-agent AlibabaCloud-Agent-Skills ``` **Python SDK**: ```python from alibabacloud_dataworks_public20240518.models import GetWorkflowDefinitionRequest request = GetWorkflowDefinitionRequest( project_id={{project_id}}, id='{{workflow_id}}' ) response = client.get_workflow_definition(request) ``` ### List Workflows **aliyun CLI**: ```bash aliyun dataworks-public ListWorkflowDefinitions \ --ProjectId {{project_id}} \ --Type CycleWorkflow \ --PageNumber 1 \ --PageSize 100 \ --user-agent AlibabaCloud-Agent-Skills ``` **Python SDK**: ```python from alibabacloud_dataworks_public20240518.models import ListWorkflowDefinitionsRequest request = ListWorkflowDefinitionsRequest( project_id={{project_id}}, type='CycleWorkflow', page_number=1, page_size=100 ) response = client.list_workflow_definitions(request) ``` ## Resource File Operations ### Create Resource **aliyun CLI**: ```bash $PYTHON $SKILL/scripts/build.py ./my_resource > /tmp/res.json aliyun dataworks-public CreateResource \ --ProjectId {{project_id}} \ --Spec "$(cat /tmp/res.json)" \ --user-agent AlibabaCloud-Agent-Skills ``` **Python SDK**: ```python from alibabacloud_dataworks_public20240518.models import CreateResourceRequest request = CreateResourceRequest( project_id={{project_id}}, spec=spec ) response = client.create_resource(request) print(f"ResourceId: {response.body.id}") ``` ### List Resources **aliyun CLI**: ```bash aliyun dataworks-public ListResources \ --ProjectId {{project_id}} \ --PageNumber 1 \ --PageSize 100 \ --user-agent AlibabaCloud-Agent-Skills ``` ## Function Operations ### Create Function **aliyun CLI**: ```bash $PYTHON $SKILL/scripts/build.py ./my_func > /tmp/func.json aliyun dataworks-public CreateFunction \ --ProjectId {{project_id}} \ --Spec "$(cat /tmp/func.json)" \ --user-agent AlibabaCloud-Agent-Skills ``` **Python SDK**: ```python from alibabacloud_dataworks_public20240518.models import CreateFunctionRequest request = CreateFunctionRequest( project_id={{project_id}}, spec=spec ) response = client.create_function(request) print(f"FunctionId: {response.body.id}") ``` ### List Functions **aliyun CLI**: ```bash aliyun dataworks-public ListFunctions \ --ProjectId {{project_id}} \ --PageNumber 1 \ --PageSize 100 \ --user-agent AlibabaCloud-Agent-Skills ``` ## Node Dependency Configuration ### Dependency Configuration When configuring inter-node dependencies, there is no need to dual-write `inputs.nodeOutputs`; only maintain dependencies in the `spec.dependencies` array: - Upstream nodes must declare `outputs.nodeOutputs` (`${projectIdentifier}.node_name`) - Downstream nodes reference upstream outputs in `spec.dependencies` - `spec.dependencies[*].nodeId` must exactly match the corresponding node's `id`, otherwise dependencies will not be recognized ```json { "spec": { "nodes": [{ "name": "downstream_node" }], "dependencies": [{ "nodeId": "downstream_node", "depends": [{ "type": "Normal", "output": "upstream_project.upstream_node_output" }] }] } } ``` ## Deployment Process Deployment is an asynchronous multi-stage pipeline. For the complete process and detailed instructions, see [deploy-guide.md](deploy-guide.md). Below are the API call templates. ### Create Deployment (Online) **Python SDK**: ```python from alibabacloud_dataworks_public20240518.models import CreatePipelineRunRequest # type: Online (deploy) or Offline (take offline) # object_ids: Only the first entity and its child entities are processed request = CreatePipelineRunRequest( project_id={{project_id}}, type='Online', object_ids=['{{object_id}}'] ) response = client.create_pipeline_run(request) run_id = response.body.id print(f"PipelineRunId: {run_id}") ``` ### Query Deployment Status **Python SDK**: ```python from alibabacloud_dataworks_public20240518.models import GetPipelineRunRequest response = client.get_pipeline_run(GetPipelineRunRequest( project_id={{project_id}}, id='{{pipeline_run_id}}' )) pipeline = response.body.pipeline.to_map() print(f"Status: {pipeline['Status']}") # Status: Init / Running / Success / Fail / Termination / Cancel for stage in pipeline.get('Stages', []): print(f" {stage['Code']}({stage['Status']}): {stage['Name']}") ``` ### Advance Deployment Stage **Python SDK**: ```python from alibabacloud_dataworks_public20240518.models import ExecPipelineRunStageRequest # code: Stage code, obtained from Stages[].Code returned by GetPipelineRun # Must advance in order; stages cannot be skipped # Async trigger; continue polling to confirm results client.exec_pipeline_run_stage(ExecPipelineRunStageRequest( project_id={{project_id}}, id='{{pipeline_run_id}}', code='{{stage_code}}' # e.g., PROD_CHECK, PROD )) ``` ### View Deployment Items **Python SDK**: ```python from alibabacloud_dataworks_public20240518.models import ListPipelineRunItemsRequest response = client.list_pipeline_run_items(ListPipelineRunItemsRequest( project_id={{project_id}}, pipeline_run_id='{{pipeline_run_id}}', page_number=1, page_size=50 )) for item in response.body.paging_info.pipeline_run_items: m = item.to_map() print(f"{m['Name']}: {m.get('Status', 'N/A')}") ``` ### Query Deployment History **Python SDK**: ```python from alibabacloud_dataworks_public20240518.models import ListPipelineRunsRequest response = client.list_pipeline_runs(ListPipelineRunsRequest( project_id={{project_id}}, page_number=1, page_size=20 )) for run in response.body.paging_info.pipeline_runs: m = run.to_map() print(f"{m['Id']} [{m['Status']}]") ``` ### Cancel Deployment **Python SDK**: ```python from alibabacloud_dataworks_public20240518.models import AbolishPipelineRunRequest client.abolish_pipeline_run(AbolishPipelineRunRequest( project_id={{project_id}}, id='{{pipeline_run_id}}' )) ``` ## Helper Queries ### Get Project Information (Convert Between projectId and projectIdentifier) **aliyun CLI**: ```bash # Get projectId by projectIdentifier aliyun dataworks-public GetProject \ --ProjectIdentifier my_project_name \ --user-agent AlibabaCloud-Agent-Skills ``` **Python SDK**: ```python from alibabacloud_dataworks_public20240518.models import GetProjectRequest request = GetProjectRequest( project_identifier='my_project_name' ) response = client.get_project(request) print(f"ProjectId: {response.body.id}") print(f"ProjectIdentifier: {response.body.project_identifier}") ``` ### List Data Sources **aliyun CLI**: ```bash aliyun dataworks-public ListDataSources \ --ProjectId {{project_id}} \ --Type odps \ --PageNumber 1 \ --PageSize 100 \ --user-agent AlibabaCloud-Agent-Skills ``` **Python SDK**: ```python from alibabacloud_dataworks_public20240518.models import ListDataSourcesRequest request = ListDataSourcesRequest( project_id={{project_id}}, type='odps', page_number=1, page_size=100 ) response = client.list_data_sources(request) for ds in response.body.paging_info.data_sources: print(f"{ds.name}: {ds.type}") ``` ### List Resource Groups **aliyun CLI**: ```bash aliyun dataworks-public ListResourceGroups \ --ProjectId {{project_id}} \ --user-agent AlibabaCloud-Agent-Skills ``` **Python SDK**: ```python from alibabacloud_dataworks_public20240518.models import ListResourceGroupsRequest request = ListResourceGroupsRequest( project_id={{project_id}} ) response = client.list_resource_groups(request) for rg in response.body.resource_groups: print(f"{rg.identifier}: {rg.name}") ``` FILE:references/cli-installation-guide.md # Aliyun CLI Installation & Configuration Guide Complete guide for installing and configuring Aliyun CLI. > **Aliyun CLI 3.3.1+**: Supports installing and using all published Alibaba Cloud product plugins. Make sure to upgrade to 3.3.1 or later for full plugin ecosystem coverage. ## Installation ### macOS **Using Homebrew (Recommended)** ```bash brew install aliyun-cli # Upgrade to latest brew upgrade aliyun-cli # Verify version (>= 3.3.1) aliyun version ``` **Using Binary** ```bash # Download wget https://aliyuncli.alicdn.com/aliyun-cli-macosx-latest-amd64.tgz # Extract tar -xzf aliyun-cli-macosx-latest-amd64.tgz # Move to PATH sudo mv aliyun /usr/local/bin/ # Verify aliyun version ``` ### Linux **Debian/Ubuntu** ```bash # Download wget https://aliyuncli.alicdn.com/aliyun-cli-linux-latest-amd64.tgz # Extract and install tar -xzf aliyun-cli-linux-latest-amd64.tgz sudo mv aliyun /usr/local/bin/ # Verify aliyun version ``` **CentOS/RHEL** ```bash # Download wget https://aliyuncli.alicdn.com/aliyun-cli-linux-latest-amd64.tgz # Extract and install tar -xzf aliyun-cli-linux-latest-amd64.tgz sudo mv aliyun /usr/local/bin/ # Verify aliyun version ``` **ARM64 Architecture** ```bash # Download ARM64 version wget https://aliyuncli.alicdn.com/aliyun-cli-linux-latest-arm64.tgz # Extract and install tar -xzf aliyun-cli-linux-latest-arm64.tgz sudo mv aliyun /usr/local/bin/ ``` ### Windows **Using Binary** 1. Download from: https://aliyuncli.alicdn.com/aliyun-cli-windows-latest-amd64.zip 2. Extract the ZIP file 3. Add the directory to your PATH environment variable 4. Open new Command Prompt or PowerShell 5. Verify: `aliyun version` **Using PowerShell** ```powershell # Download Invoke-WebRequest -Uri "https://aliyuncli.alicdn.com/aliyun-cli-windows-latest-amd64.zip" -OutFile "aliyun-cli.zip" # Extract Expand-Archive -Path aliyun-cli.zip -DestinationPath C:\aliyun-cli # Add to PATH (requires admin privileges) $env:Path += ";C:\aliyun-cli" [Environment]::SetEnvironmentVariable("Path", $env:Path, [System.EnvironmentVariableTarget]::Machine) # Verify aliyun version ``` ## Configuration ### Quick Start ```bash aliyun configure set \ --mode AK \ --access-key-id \ --access-key-secret \ --region cn-hangzhou ``` All `aliyun configure` commands support non-interactive flags, which is the recommended approach — it works in scripts, CI/CD pipelines, and agent-driven automation without hanging on stdin prompts. **Where to Get Access Keys** 1. Log in to Aliyun Console: https://ram.console.aliyun.com/ 2. Navigate to: AccessKey Management 3. Create a new AccessKey pair 4. Save the secret immediately — it's only shown once ### Configuration Modes Aliyun CLI supports 6 authentication modes. All examples below use non-interactive flags. #### 1. AK Mode (Access Key) Most common mode for personal accounts and scripts. ```bash aliyun configure set \ --mode AK \ --access-key-id LTAI5tXXXXXXXX \ --access-key-secret 8dXXXXXXXXXXXXXXXXXXXXXXXX \ --region cn-hangzhou ``` Configuration is stored in `~/.aliyun/config.json`: ```json { "current": "default", "profiles": [ { "name": "default", "mode": "AK", "access_key_id": "LTAI5tXXXXXXXX", "access_key_secret": "8dXXXXXXXXXXXXXXXXXXXXXXXX", "region_id": "cn-hangzhou", "output_format": "json", "language": "en" } ] } ``` #### 2. StsToken Mode (Temporary Credentials) For short-lived access (tokens expire in 1-12 hours). ```bash aliyun configure set \ --mode StsToken \ --access-key-id LTAI5tXXXXXXXX \ --access-key-secret 8dXXXXXXXXXXXXXXXXXXXXXXXX \ --sts-token v1.0:XXXXXXXXXXXXXXXX \ --region cn-hangzhou ``` Use cases: CI/CD pipelines, temporary access for external contractors, cross-account access. #### 3. RamRoleArn Mode (Assume RAM Role) Assume a RAM role for elevated or cross-account access. ```bash aliyun configure set \ --mode RamRoleArn \ --access-key-id LTAI5tXXXXXXXX \ --access-key-secret 8dXXXXXXXXXXXXXXXXXXXXXXXX \ --ram-role-arn acs:ram::123456789012:role/AdminRole \ --role-session-name my-session \ --region cn-hangzhou ``` Use cases: cross-account resource access, temporary elevated privileges, role-based access control. #### 4. EcsRamRole Mode (ECS Instance RAM Role) Use the RAM role attached to an ECS instance — no credentials needed. ```bash aliyun configure set \ --mode EcsRamRole \ --ram-role-name MyEcsRole \ --region cn-hangzhou ``` Requirements: must be running on an ECS instance with a RAM role attached. Use cases: scripts and automation running on ECS instances. #### 5. RsaKeyPair Mode (RSA Key Pair) Use RSA key pair for authentication (generate key pair in Aliyun Console first). ```bash aliyun configure set \ --mode RsaKeyPair \ --private-key /path/to/private-key.pem \ --key-pair-name my-key-pair \ --region cn-hangzhou ``` #### 6. RamRoleArnWithEcs Mode (ECS + RAM Role) Combine ECS instance role with RAM role assumption for cross-account access from ECS. ```bash aliyun configure set \ --mode RamRoleArnWithEcs \ --ram-role-name MyEcsRole \ --ram-role-arn acs:ram::123456789012:role/TargetRole \ --role-session-name my-session \ --region cn-hangzhou ``` ### Environment Variables **Highest priority** - overrides config file **Access Key Mode** ```bash export ALIBABA_CLOUD_ACCESS_KEY_ID=your_access_key_id export ALIBABA_CLOUD_ACCESS_KEY_SECRET=your_access_key_secret export ALIBABA_CLOUD_REGION_ID=cn-hangzhou ``` **STS Token Mode** ```bash export ALIBABA_CLOUD_ACCESS_KEY_ID=your_access_key_id export ALIBABA_CLOUD_ACCESS_KEY_SECRET=your_access_key_secret export ALIBABA_CLOUD_SECURITY_TOKEN=your_sts_token export ALIBABA_CLOUD_REGION_ID=cn-hangzhou ``` **ECS RAM Role Mode** ```bash export ALIBABA_CLOUD_ECS_METADATA=role_name ``` **Use Case**: - CI/CD pipelines - Docker containers - Temporary credential override ### Managing Multiple Profiles **Create Named Profiles** ```bash aliyun configure set --profile projectA \ --mode AK \ --access-key-id LTAI5tAAAAAAAA \ --access-key-secret 8dAAAAAAAAAAAAAAAAAAAAAAAA \ --region cn-hangzhou aliyun configure set --profile projectB \ --mode AK \ --access-key-id LTAI5tBBBBBBBB \ --access-key-secret 8dBBBBBBBBBBBBBBBBBBBBBBBB \ --region cn-shanghai ``` **Use Specific Profile** ```bash aliyun ecs describe-instances --profile projectA export ALIBABA_CLOUD_PROFILE=projectA aliyun ecs describe-instances # Uses projectA ``` **List and Switch Profiles** ```bash aliyun configure list # List all profiles aliyun configure set --current projectA # Switch default profile ``` ### Credential Priority Credentials are loaded in this order (first found wins): 1. **Command-line flag**: `--profile ` 2. **Environment variable**: `ALIBABA_CLOUD_PROFILE` 3. **Environment credentials**: `ALIBABA_CLOUD_ACCESS_KEY_ID`, etc. 4. **Configuration file**: `~/.aliyun/config.json` (current profile) 5. **ECS Instance RAM Role**: If running on ECS with attached role ## Verification ### Test Authentication ```bash # Basic test - list regions aliyun ecs describe-regions # Expected output: JSON array of regions ``` **If successful**, you'll see: ```json { "Regions": { "Region": [ { "RegionId": "cn-hangzhou", "RegionEndpoint": "ecs.cn-hangzhou.aliyuncs.com", "LocalName": "China (Hangzhou)" }, ... ] }, "RequestId": "..." } ``` **If failed**, you'll see error messages: - `InvalidAccessKeyId.NotFound` - Wrong Access Key ID - `SignatureDoesNotMatch` - Wrong Access Key Secret - `InvalidSecurityToken.Expired` - STS token expired (for StsToken mode) - `Forbidden.RAM` - Insufficient permissions ### Debug Configuration ```bash # Show current configuration aliyun configure get # Test with debug logging aliyun ecs describe-regions --log-level=debug # Check credential provider aliyun configure get mode ``` ## Security Best Practices ### 1. Use RAM Users (Not Root Account) ❌ **Don't**: Use Aliyun root account credentials ✅ **Do**: Create RAM users with specific permissions ```bash # Create RAM user in console # Attach only necessary policies # Use RAM user's access keys ``` ### 2. Principle of Least Privilege Grant only the minimum permissions needed: ```bash # Example: Read-only ECS access # Attach policy: AliyunECSReadOnlyAccess ``` ### 3. Rotate Access Keys Regularly ```bash # Create new access key in RAM Console, then update configuration aliyun configure set --access-key-id NEW_KEY --access-key-secret NEW_SECRET # Delete old access key from console ``` ### 4. Use STS Tokens for Temporary Access ```bash aliyun configure set --mode StsToken \ --access-key-id XXXX --access-key-secret XXXX \ --sts-token XXXX --region cn-hangzhou ``` ### 5. Use ECS RAM Roles When Possible ```bash aliyun configure set --mode EcsRamRole --ram-role-name MyRole --region cn-hangzhou ``` ### 6. Never Commit Credentials ```bash # Add to .gitignore echo "~/.aliyun/config.json" >> .gitignore # Use environment variables in CI/CD instead ``` ### 7. Secure Config File ```bash # Restrict permissions chmod 600 ~/.aliyun/config.json ``` ## Troubleshooting ### Issue: Command Not Found ```bash # Check installation which aliyun # Check PATH echo $PATH # Reinstall or add to PATH ``` ### Issue: Authentication Failed ```bash # Verify configuration aliyun configure get # Test with debug aliyun ecs describe-regions --log-level=debug # Check credentials in console # Verify access key is active ``` ### Issue: Permission Denied ```bash # Error: Forbidden.RAM # Check RAM user permissions # Attach necessary policies in RAM console # Example: AliyunECSFullAccess for ECS operations ``` ### Issue: STS Token Expired ```bash # Error: InvalidSecurityToken.Expired # Reconfigure with new token aliyun configure set --mode StsToken \ --access-key-id XXXX --access-key-secret XXXX \ --sts-token NEW_TOKEN --region cn-hangzhou ``` ### Issue: Wrong Region ```bash # Some resources may not exist in the specified region # Check available regions aliyun ecs describe-regions # Update default region aliyun configure set region cn-shanghai ``` ## Advanced Configuration ### Custom Endpoint ```bash # Use custom or private endpoint export ALIBABA_CLOUD_ECS_ENDPOINT=ecs-vpc.cn-hangzhou.aliyuncs.com ``` ### Proxy Settings ```bash # HTTP proxy export HTTP_PROXY=http://proxy.example.com:8080 export HTTPS_PROXY=http://proxy.example.com:8080 # No proxy for specific domains export NO_PROXY=localhost,127.0.0.1,.aliyuncs.com ``` ### Timeout Settings ```bash # Connection timeout (default: 10s) export ALIBABA_CLOUD_CONNECT_TIMEOUT=30 # Read timeout (default: 10s) export ALIBABA_CLOUD_READ_TIMEOUT=30 ``` ## Next Steps After installation and configuration: 1. **Install plugins** for services you need (v3.3.1+ supports all published product plugins): ```bash aliyun plugin install --names ecs vpc rds # List all available plugins aliyun plugin list-remote ``` 2. **Explore commands**: ```bash aliyun ecs --help aliyun fc --help ``` 3. **Read documentation**: - [Command Syntax Guide](./command-syntax.md) - [Global Flags Reference](./global-flags.md) - [Common Scenarios](./common-scenarios.md) ## References - Official Documentation: https://help.aliyun.com/zh/cli/ - RAM Console: https://ram.console.aliyun.com/ - Access Key Management: https://ram.console.aliyun.com/manage/ak - Plugin Repository: https://github.com/aliyun/aliyun-cli FILE:references/deploy-guide.md # Deployment Guide This document provides a detailed description of the deployment (online/offline) process in DataWorks data development. Deployment is an asynchronous multi-stage pipeline that requires polling for status and advancing each stage. --- ## Workspace Modes and Deployment DataWorks workspaces have two modes with different deployment processes: | | Simple Mode | Standard Mode | |---|---|---| | Environments | Production only | Development + Production | | Development flow | Create node -> Deploy -> Production scheduling | Create node (dev) -> Submit -> Deploy -> Production scheduling | | Deployment meaning | Deploy directly to production | Deploy from development to production environment | | Number of stages | Fewer (3 observed in practice) | More (may include code review, smoke test, approval, etc.) | **How to determine**: Use the `GetProject` API to check `envTypes`; size=1 means Simple Mode, size=2 means Standard Mode. --- ## Deployment State Machine ### Pipeline Overall Status ``` Init -> Running -> Success -> Fail -> Termination -> Cancel ``` | Status | Meaning | |------|------| | `Init` | Initialized | | `Running` | In progress (includes waiting for stage advancement) | | `Success` | All stages completed, deployment successful | | `Fail` | A stage has failed | | `Termination` | Terminated | | `Cancel` | Cancelled | ### Stage Status Each stage has the same status values as the Pipeline: `Init` / `Running` / `Success` / `Fail` / `Termination` / `Cancel` ### Stage Types | Type | Meaning | Requires manual advancement | |------|------|:---:| | `Build` | Build the deployment package | No (runs automatically) | | `Check` | Check (production checker, code review, etc.) | Yes | | `Deploy` | Deploy to production environment | Yes | | `Offline` | Take offline | Yes | | `Delete` | Delete | Yes | ### Observed Stages in Simple Mode ``` Step 1: BUILD_PACKAGE (Type=Build) <- Runs automatically Step 2: PROD_CHECK (Type=Check) <- Requires ExecPipelineRunStage to advance Step 3: PROD (Type=Deploy) <- Requires ExecPipelineRunStage to advance ``` BUILD_PACKAGE includes the following checks: - `BuildPackageChecker` -- Package build check - `NodeParentDependency` -- Downstream dependency check - `NodeInProcess` -- In-progress offline process check ### Possible Stages in Standard Mode Stages in Standard Mode are based on the actual response from `GetPipelineRun` and may include: - `DEV_CHECK` -- Development environment check - Code review stage - Smoke test stage - Approval stage (may require manual action in the console) - `PROD_CHECK` -- Production environment check - `PROD` -- Deploy to production **Do not hardcode the stage list**; always handle stages dynamically based on the Stages returned by the API. --- ## Deployment API Overview | API | Purpose | |-----|------| | `CreatePipelineRun` | Create a deployment process | | `GetPipelineRun` | Get deployment status and stages | | `ExecPipelineRunStage` | Advance a stage (async) | | `ListPipelineRunItems` | View the list of nodes included in the deployment | | `ListPipelineRuns` | Query deployment history | | `AbolishPipelineRun` | Cancel a deployment | --- ## CLI vs SDK Response Field Differences > **Important**: The `aliyun` CLI and Python SDK return different JSON structures; do not mix them. | Scenario | CLI (`aliyun` command) | Python SDK | |------|---------------------|------------| | Deployment creation returns ID | `json['Id']` | `resp.body.id` | | Get Pipeline object | `json['Pipeline']` | `resp.body.pipeline.to_map()` | | Pipeline status | `json['Pipeline']['Status']` | `pipeline['Status']` | | Stages list | `json['Pipeline']['Stages']` | `pipeline['Stages']` | --- ## Deployment Process (Online) ### Step 1: Create Deployment **CLI**: ```bash aliyun dataworks-public CreatePipelineRun \ --ProjectId $PROJECT_ID \ --Type Online \ --ObjectIds "[\"$WORKFLOW_ID\"]" \ --user-agent AlibabaCloud-Agent-Skills # Example response: {"Id": "ae781cc7-...", "RequestId": "..."} # Record the Id for subsequent polling ``` **SDK**: ```python from alibabacloud_dataworks_public20240518.models import CreatePipelineRunRequest resp = client.create_pipeline_run(CreatePipelineRunRequest( project_id=PROJECT_ID, type='Online', object_ids=['node_ID_or_workflow_ID'] )) run_id = resp.body.id ``` **Notes**: - `type` values: `Online` (deploy) or `Offline` (take offline) - `object_ids` only processes the first entity and its child entities; batch deployment of multiple independent entities is not supported - When deploying a workflow, pass the workflow ID to deploy all internal nodes simultaneously ### Step 2: Poll and Advance Each Stage **CLI** (copy-ready): ```bash #!/bin/bash # Deployment polling and advancement script (CLI version) PIPELINE_ID="" PROJECT_ID="" for i in $(seq 1 60); do RESP=$(aliyun dataworks-public GetPipelineRun \ --Id "$PIPELINE_ID" --ProjectId "$PROJECT_ID" \ --user-agent AlibabaCloud-Agent-Skills 2>&1) # Note: CLI returns Pipeline as the top-level key (not PipelineRun) STATUS=$(echo "$RESP" | python3 -c "import sys,json; print(json.load(sys.stdin).get('Pipeline',{}).get('Status',''))") echo "[$i] Pipeline status: $STATUS" # Terminal state check if [ "$STATUS" = "Success" ] || [ "$STATUS" = "Fail" ] || [ "$STATUS" = "Cancel" ]; then echo "Pipeline finished: $STATUS" break fi # Find the Init stage that needs to be advanced (all prior stages are Success) STAGE_CODE=$(echo "$RESP" | python3 -c " import sys, json data = json.load(sys.stdin).get('Pipeline', {}) stages = data.get('Stages', []) for s in stages: if s.get('Status') == 'Init': prior_ok = all(s2.get('Status') == 'Success' for s2 in stages if s2.get('Step',0) < s.get('Step',0)) if prior_ok: print(s.get('Code', '')) break ") if [ -n "$STAGE_CODE" ]; then echo " Pushing stage: $STAGE_CODE" aliyun dataworks-public ExecPipelineRunStage \ --Id "$PIPELINE_ID" --ProjectId "$PROJECT_ID" --Code "$STAGE_CODE" \ --user-agent AlibabaCloud-Agent-Skills fi sleep 5 done ``` **SDK**: ```python import time from alibabacloud_dataworks_public20240518.models import ( GetPipelineRunRequest, ExecPipelineRunStageRequest ) MAX_POLL = 60 # Maximum number of polling attempts POLL_INTERVAL = 3 # Polling interval (seconds) for i in range(MAX_POLL): time.sleep(POLL_INTERVAL) resp = client.get_pipeline_run(GetPipelineRunRequest( project_id=PROJECT_ID, id=run_id )) pipeline = resp.body.pipeline.to_map() status = pipeline['Status'] stages = pipeline.get('Stages', []) # Print current status stage_info = ' -> '.join(f"{s['Code']}({s['Status']})" for s in stages) print(f"[{status}] {stage_info}") # Terminal state check if status in ('Success', 'Fail', 'Termination', 'Cancel'): if status == 'Success': print("Deployment successful") else: msg = pipeline.get('Message', '') print(f"Deployment ended: {status}, {msg}") break # Find the stage that needs to be advanced for j, stage in enumerate(stages): if stage['Status'] == 'Fail': print(f"Stage failed: {stage['Name']} - {stage.get('Message', '')}") break if stage['Status'] == 'Init': # Check if all prior stages have completed prev_all_success = all( stages[k]['Status'] == 'Success' for k in range(j) ) if prev_all_success: print(f"Advancing stage: {stage['Name']} ({stage['Code']})") try: client.exec_pipeline_run_stage(ExecPipelineRunStageRequest( project_id=PROJECT_ID, id=run_id, code=stage['Code'] )) except Exception as e: print(f"Advancement failed: {e}") break # Process only one stage at a time ``` ### Step 3: Confirm Deployment Result ```python from alibabacloud_dataworks_public20240518.models import ListPipelineRunItemsRequest resp = client.list_pipeline_run_items(ListPipelineRunItemsRequest( project_id=PROJECT_ID, pipeline_run_id=run_id, page_number=1, page_size=50 )) for item in resp.body.paging_info.pipeline_run_items: m = item.to_map() print(f"{m['Name']}: {m.get('Status', 'N/A')}") ``` --- ## Offline Process The offline process uses `type` `Offline`. **Stages differ from the online process** (4 stages observed in practice): ``` Online Stages: BUILD_PACKAGE(Build) -> PROD_CHECK(Check) -> PROD(Deploy) Offline Stages: OfflineCheck(Check) -> PROD_CHECK(Check) -> PROD(Offline) ``` | Stage | Type | Description | |-------|------|------| | `OfflineCheck` | Check | Pre-offline check (runs automatically) | | `PROD_CHECK` | Check | Production checker (requires advancement) | | `PROD` | Offline | Remove from production scheduling (requires advancement) | ```python resp = client.create_pipeline_run(CreatePipelineRunRequest( project_id=PROJECT_ID, type='Offline', object_ids=['node_ID_or_workflow_ID'] )) run_id = resp.body.id # Subsequent polling and advancement is the same as the online process, but note the different stages ``` The polling and advancement logic is identical to the online process (see Step 2). The Agent does not need to differentiate between online/offline stage codes; simply advance dynamically based on the Stages returned by `GetPipelineRun`. --- ## Cancel Deployment If a deployment is stuck or needs to be revoked: ```python from alibabacloud_dataworks_public20240518.models import AbolishPipelineRunRequest client.abolish_pipeline_run(AbolishPipelineRunRequest( project_id=PROJECT_ID, id=run_id )) ``` --- ## Query Deployment History ```python from alibabacloud_dataworks_public20240518.models import ListPipelineRunsRequest resp = client.list_pipeline_runs(ListPipelineRunsRequest( project_id=PROJECT_ID, page_number=1, page_size=20 )) for run in resp.body.paging_info.pipeline_runs: m = run.to_map() stages = ' -> '.join(f"{s['Code']}({s['Status']})" for s in m.get('Stages', [])) print(f"{m['Id']} [{m['Status']}] {stages}") ``` You can filter by `status` (e.g., only Running deployments) or by `object_id` (deployment history for a specific node). --- ## FAQ ### 1. Deployment stuck at PROD_CHECK or a Check stage **Cause**: Check-type stages do not execute automatically; they require `ExecPipelineRunStage` to advance. **Solution**: Ensure the polling logic includes advancement logic -- when an `Init` stage is encountered and all prior stages are `Success`, advance it automatically. ### 2. ExecPipelineRunStage returns success but stage status doesn't change **Cause**: `ExecPipelineRunStage` is an async trigger; the response only indicates successful triggering, not stage completion. **Solution**: Continue polling and wait for the stage status to change from `Init` to `Running` and then to `Success`. ### 3. Approval stage cannot be advanced via API **Symptom**: Calling `ExecPipelineRunStage` returns an error, or the stage status remains unchanged. **Cause**: In Standard Mode, certain stages (such as approval) require a user with specific permissions to operate manually in the DataWorks console. **Solution**: The Agent should recognize this situation and inform the user: "The current deployment is awaiting approval. Please approve it on the Task Deployment page in the DataWorks console." After approval is completed, continue polling and advancing subsequent stages. ### 4. Deployment failed with error in Stage.Message **Solution**: Read the `Message` field of the failed stage. Common causes include: - Node code syntax errors - Upstream dependency node not yet deployed - Incorrect resource group configuration - Insufficient permissions ### 5. Passed multiple object_ids but only the first one took effect **Cause**: The official `CreatePipelineRun` documentation states "only the first entity in the array and its child entities will be successfully deployed." **Solution**: To deploy multiple independent entities, create separate PipelineRuns. If the nodes are within the same workflow, simply deploy the workflow ID to include all child nodes. ### 6. Deployment is stuck and needs to be cancelled **Solution**: Call `AbolishPipelineRun` to cancel. After cancellation, the Pipeline status changes to `Cancel`. ### 7. Deployment during the bulk instance generation window **Note**: The period from 23:30 to 24:00 each day is the bulk instance generation window. Deployments during this period will not take effect until the **third day** after the operation, not the next day. Avoid deploying during this window. FILE:references/di-guide.md # DI Data Synchronization Development Guide DI (Data Integration) is DataWorks' offline data synchronization service that supports data migration and synchronization across various heterogeneous data sources. This document describes the DI node code format, configuration methods, and common scenarios. ## DI Node Overview DI node type identifiers in FlowSpec: | Field | Value | |------|------| | `script.runtime.command` | `DI` | | `script.language` | `di` | | Code file extension | `.json` | | `datasourceType` | `null` (no node-level datasource needed) | The DI node's code file is a JSON-formatted DIJob definition that describes the complete configuration for data reading (Reader) and writing (Writer). --- ## DIJob JSON Structure Top-level structure of the DI code file: ```json { "type": "job", "version": "2.0", "steps": [ { "stepType": "mysql", "name": "Reader", "category": "reader", "parameter": { ... } }, { "stepType": "odps", "name": "Writer", "category": "writer", "parameter": { ... } } ], "order": { "hops": [ { "from": "Reader", "to": "Writer" } ] }, "setting": { "speed": { "concurrent": 3, "throttle": false }, "errorLimit": { "record": 0 } } } ``` ### Top-Level Fields | Field | Type | Required | Description | |------|------|------|------| | `type` | string | Yes | Fixed value `"job"` | | `version` | string | Yes | Version number, recommended `"2.0"` | | `steps` | array | Yes | Step array, must contain at least one Reader and one Writer | | `order` | object | Yes | Step execution order | | `setting` | object | No | Runtime parameter configuration | --- ## steps Configuration The `steps` array contains two types of steps: Reader (reads source data) and Writer (writes to target). ### Step Common Fields | Field | Type | Required | Description | |------|------|------|------| | `stepType` | string | Yes | Data source type identifier (e.g., `mysql`, `odps`, `oss`, `hologres`) | | `name` | string | Yes | Step name, referenced in `order.hops` | | `category` | string | Yes | Step category: `"reader"` or `"writer"` | | `parameter` | object | Yes | Step parameters, varies by data source type | --- ## Reader Configuration Details ### MySQL Reader Reads data from a MySQL database. ```json { "stepType": "mysql", "name": "Reader", "category": "reader", "parameter": { "datasource": "my_mysql_datasource", "column": ["id", "name", "age", "created_at"], "connection": [ { "table": ["user_info"], "datasource": "my_mysql_datasource" } ], "where": "created_at >= '${bizdate} 00:00:00'", "splitPk": "id", "encoding": "UTF-8" } } ``` | Parameter | Type | Required | Description | |------|------|------|------| | `datasource` | string | Yes | Datasource name (a MySQL datasource registered in DataWorks) | | `column` | array[string] | Yes | List of column names to read | | `connection` | array | Yes | Connection configuration, contains table name and datasource | | `connection[].table` | array[string] | Yes | Table name list | | `connection[].datasource` | string | Yes | Datasource name | | `where` | string | No | Filter condition (SQL WHERE clause) | | `splitPk` | string | No | Split key for concurrent reads, recommend using the primary key | | `encoding` | string | No | Character encoding, default `UTF-8` | ### MaxCompute (ODPS) Reader Reads data from a MaxCompute table. ```json { "stepType": "odps", "name": "Reader", "category": "reader", "parameter": { "datasource": "odps_first", "table": "ods_user_info", "column": [ {"name": "id", "type": "bigint"}, {"name": "name", "type": "string"}, {"name": "age", "type": "bigint"} ], "partition": "dt=${bizdate}" } } ``` | Parameter | Type | Required | Description | |------|------|------|------| | `datasource` | string | Yes | MaxCompute datasource name | | `table` | string | Yes | Table name | | `column` | array | Yes | Column definitions, can be strings (column names) or objects (with name and type) | | `partition` | string | No | Partition filter condition (e.g., `dt=20260321`) | | `where` | string | No | Filter condition | ### OSS Reader Reads data from OSS object storage. ```json { "stepType": "oss", "name": "Reader", "category": "reader", "parameter": { "datasource": "my_oss_datasource", "object": ["data/input/user_info_${bizdate}.csv"], "column": [ {"type": "long", "index": 0}, {"type": "string", "index": 1}, {"type": "long", "index": 2}, {"type": "date", "index": 3} ], "fieldDelimiter": ",", "encoding": "UTF-8", "fileFormat": "csv" } } ``` | Parameter | Type | Required | Description | |------|------|------|------| | `datasource` | string | Yes | OSS datasource name | | `object` | array[string] | Yes | File path list (supports wildcards) | | `column` | array | Yes | Column definitions, with `type` (data type) and `index` (column position, starting from 0) | | `fieldDelimiter` | string | No | Field delimiter | | `encoding` | string | No | Character encoding, default `UTF-8` | | `fileFormat` | string | No | File format: `csv`, `text`, `parquet`, `orc`, `json` | ### Hologres Reader Reads data from Hologres. ```json { "stepType": "hologres", "name": "Reader", "category": "reader", "parameter": { "datasource": "my_holo_datasource", "table": "public.user_info", "column": ["id", "name", "age"] } } ``` ### PostgreSQL Reader Reads data from PostgreSQL. ```json { "stepType": "postgresql", "name": "Reader", "category": "reader", "parameter": { "datasource": "my_pg_datasource", "column": ["id", "name", "age"], "connection": [ { "table": ["user_info"], "datasource": "my_pg_datasource" } ], "where": "created_at >= '${bizdate}'" } } ``` --- ## Writer Configuration Details ### MaxCompute (ODPS) Writer Writes to a MaxCompute table. ```json { "stepType": "odps", "name": "Writer", "category": "writer", "parameter": { "datasource": "odps_first", "table": "dwd_user_info", "column": [ {"name": "id", "type": "bigint"}, {"name": "name", "type": "string"}, {"name": "age", "type": "bigint"} ], "partition": "dt=${bizdate}", "truncate": true } } ``` | Parameter | Type | Required | Description | |------|------|------|------| | `datasource` | string | Yes | MaxCompute datasource name | | `table` | string | Yes | Target table name | | `column` | array | Yes | Column definitions, can be strings or objects (with name and type) | | `partition` | string | No | Target partition (e.g., `dt=20260321`) | | `truncate` | boolean | No | Whether to clear the partition/table first, default `true` | ### Hologres Writer Writes to a Hologres table. ```json { "stepType": "hologres", "name": "Writer", "category": "writer", "parameter": { "datasource": "my_holo_datasource", "table": "public.dwd_user_info", "column": ["id", "name", "age", "updated_at"], "writeMode": "insertOrReplace", "batchSize": 512 } } ``` | Parameter | Type | Required | Description | |------|------|------|------| | `datasource` | string | Yes | Hologres datasource name | | `table` | string | Yes | Target table name (including schema, e.g., `public.table_name`) | | `column` | array[string] | Yes | Column name list | | `writeMode` | string | No | Write mode: `insertOrIgnore` (ignore on conflict), `insertOrReplace` (overwrite on conflict), `insertOrUpdate` (update on conflict) | | `conflictMode` | string | No | Conflict handling mode | | `batchSize` | integer | No | Batch write size, default 512 | ### MySQL Writer Writes to a MySQL database. ```json { "stepType": "mysql", "name": "Writer", "category": "writer", "parameter": { "datasource": "my_mysql_datasource", "column": ["id", "name", "age", "updated_at"], "connection": [ { "table": ["target_user_info"], "datasource": "my_mysql_datasource" } ], "writeMode": "replace", "preSql": ["DELETE FROM target_user_info WHERE dt='${bizdate}'"], "postSql": [], "batchSize": 1024 } } ``` | Parameter | Type | Required | Description | |------|------|------|------| | `datasource` | string | Yes | MySQL datasource name | | `column` | array[string] | Yes | Column name list | | `connection` | array | Yes | Connection configuration, contains target table name and datasource | | `writeMode` | string | No | Write mode: `insert`, `replace`, `update` | | `preSql` | array[string] | No | SQL statements to execute before writing | | `postSql` | array[string] | No | SQL statements to execute after writing | | `batchSize` | integer | No | Batch write size, default 1024 | ### OSS Writer Writes to OSS object storage. ```json { "stepType": "oss", "name": "Writer", "category": "writer", "parameter": { "datasource": "my_oss_datasource", "object": "data/output/result_${bizdate}.csv", "column": [ {"name": "id", "type": "long"}, {"name": "name", "type": "string"}, {"name": "amount", "type": "double"} ], "writeMode": "truncate", "fieldDelimiter": ",", "encoding": "UTF-8", "fileFormat": "csv" } } ``` | Parameter | Type | Required | Description | |------|------|------|------| | `datasource` | string | Yes | OSS datasource name | | `object` | string | Yes | Output file path | | `column` | array | Yes | Column definitions, with `name` (column name) and `type` (data type) | | `writeMode` | string | No | Write mode: `truncate` (overwrite), `append`, `nonConflict` (write only when no conflict) | | `fieldDelimiter` | string | No | Field delimiter | | `encoding` | string | No | Character encoding, default `UTF-8` | | `fileFormat` | string | No | File format: `csv`, `text`, `parquet`, `orc`, `json` | --- ## setting Configuration `setting` defines the runtime parameters for the data synchronization task. ```json "setting": { "speed": { "concurrent": 3, "throttle": false, "mbps": 10, "dmu": 5 }, "errorLimit": { "record": 0 } } ``` ### speed (Speed Configuration) | Parameter | Type | Description | |------|------|------| | `concurrent` | integer | Concurrency level, i.e., the number of simultaneous data channels, minimum 1 | | `throttle` | boolean | Whether to enable throttling. When `true`, bandwidth is limited by `mbps` | | `mbps` | number | Throttle value (MB/s), only effective when `throttle` is `true` | | `dmu` | integer | Data Migration Unit count, affects resource allocation | ### errorLimit (Error Tolerance Configuration) | Parameter | Type | Description | |------|------|------| | `record` | integer | Maximum number of dirty data records allowed. `0` means no dirty data is allowed; the task will fail if the threshold is exceeded | --- ## order Configuration `order` defines the execution order between steps. ```json "order": { "hops": [ { "from": "Reader", "to": "Writer" } ] } ``` - `from`: The `name` of the source step - `to`: The `name` of the target step - For simple single-Reader single-Writer scenarios, only one hop is needed --- ## Common Data Synchronization Scenarios ### Scenario 1: MySQL to MaxCompute Synchronize MySQL business data to MaxCompute for offline analysis. ```json { "type": "job", "version": "2.0", "steps": [ { "stepType": "mysql", "name": "Reader", "category": "reader", "parameter": { "datasource": "rds_prod", "column": ["id", "user_name", "email", "status", "created_at"], "connection": [ { "table": ["t_user"], "datasource": "rds_prod" } ], "where": "created_at >= '${bizdate} 00:00:00' AND created_at < '${bizdate} 23:59:59'", "splitPk": "id" } }, { "stepType": "odps", "name": "Writer", "category": "writer", "parameter": { "datasource": "odps_first", "table": "ods_user", "column": [ {"name": "id", "type": "bigint"}, {"name": "user_name", "type": "string"}, {"name": "email", "type": "string"}, {"name": "status", "type": "bigint"}, {"name": "created_at", "type": "datetime"} ], "partition": "dt=${bizdate}", "truncate": true } } ], "order": { "hops": [{ "from": "Reader", "to": "Writer" }] }, "setting": { "speed": { "concurrent": 5, "throttle": false }, "errorLimit": { "record": 0 } } } ``` ### Scenario 2: MySQL to Hologres Synchronize MySQL data to Hologres for real-time analysis. ```json { "type": "job", "version": "2.0", "steps": [ { "stepType": "mysql", "name": "Reader", "category": "reader", "parameter": { "datasource": "rds_prod", "column": ["id", "product_name", "price", "stock"], "connection": [ { "table": ["t_product"], "datasource": "rds_prod" } ] } }, { "stepType": "hologres", "name": "Writer", "category": "writer", "parameter": { "datasource": "holo_prod", "table": "public.ods_product", "column": ["id", "product_name", "price", "stock"], "writeMode": "insertOrReplace", "batchSize": 512 } } ], "order": { "hops": [{ "from": "Reader", "to": "Writer" }] }, "setting": { "speed": { "concurrent": 3, "throttle": false }, "errorLimit": { "record": 0 } } } ``` ### Scenario 3: MaxCompute to MySQL Export MaxCompute analysis results back to MySQL for use by business systems. ```json { "type": "job", "version": "2.0", "steps": [ { "stepType": "odps", "name": "Reader", "category": "reader", "parameter": { "datasource": "odps_first", "table": "ads_user_report", "column": [ {"name": "user_id", "type": "bigint"}, {"name": "total_orders", "type": "bigint"}, {"name": "total_amount", "type": "double"}, {"name": "report_date", "type": "string"} ], "partition": "dt=${bizdate}" } }, { "stepType": "mysql", "name": "Writer", "category": "writer", "parameter": { "datasource": "rds_prod", "column": ["user_id", "total_orders", "total_amount", "report_date"], "connection": [ { "table": ["t_user_report"], "datasource": "rds_prod" } ], "writeMode": "replace", "preSql": ["DELETE FROM t_user_report WHERE report_date='${bizdate}'"], "batchSize": 1024 } } ], "order": { "hops": [{ "from": "Reader", "to": "Writer" }] }, "setting": { "speed": { "concurrent": 3, "throttle": true, "mbps": 5 }, "errorLimit": { "record": 0 } } } ``` ### Scenario 4: OSS to MaxCompute Import CSV files from OSS into MaxCompute. ```json { "type": "job", "version": "2.0", "steps": [ { "stepType": "oss", "name": "Reader", "category": "reader", "parameter": { "datasource": "oss_data_lake", "object": ["data/daily/${bizdate}/*.csv"], "column": [ {"type": "long", "index": 0}, {"type": "string", "index": 1}, {"type": "string", "index": 2}, {"type": "double", "index": 3}, {"type": "date", "index": 4} ], "fieldDelimiter": ",", "encoding": "UTF-8", "fileFormat": "csv" } }, { "stepType": "odps", "name": "Writer", "category": "writer", "parameter": { "datasource": "odps_first", "table": "ods_external_data", "column": [ {"name": "id", "type": "bigint"}, {"name": "category", "type": "string"}, {"name": "description", "type": "string"}, {"name": "amount", "type": "double"}, {"name": "event_date", "type": "datetime"} ], "partition": "dt=${bizdate}", "truncate": true } } ], "order": { "hops": [{ "from": "Reader", "to": "Writer" }] }, "setting": { "speed": { "concurrent": 3, "throttle": false }, "errorLimit": { "record": 10 } } } ``` --- ## DI Node Creation Process The DI node creation process is the same as for regular nodes; the only difference is that the code file is in JSON format. ```bash # 1. Create the node directory mkdir -p ./sync_user_to_odps # 2. Copy the node template # Refer to the DI template in assets/templates/ and modify accordingly # 3. Edit spec.json # - name: sync_user_to_odps # - script.runtime.command: DI # - script.language: di # - No need to configure datasource (DI node datasources are configured in the code JSON) # 4. Write the DI code file # Create sync_user_to_odps.json and fill in the DIJob JSON # 5. Create dataworks.properties cat > ./sync_user_to_odps/dataworks.properties << 'EOF' projectIdentifier=my_project spec.runtimeResource.resourceGroup=S_res_group_xxx EOF # 6. Build spec JSON and submit via API aliyun dataworks-public CreateNode \ --ProjectId {{project_id}} \ --Scene DATAWORKS_PROJECT \ --Spec "$(cat /tmp/spec.json)" \ --user-agent AlibabaCloud-Agent-Skills ``` --- ## Important Notes 1. **Datasource name consistency**: The datasource name in `parameter.datasource` and `connection[].datasource` must exactly match the datasource name registered in DataWorks 2. **Column count alignment**: The number of columns in Reader and Writer must be identical, with one-to-one correspondence in order 3. **Partition format**: MaxCompute partition values use the `dt=${bizdate}` format, supporting scheduling parameter substitution 4. **Concurrency setting**: Setting `concurrent` too high may put excessive load on the source database; adjust based on actual conditions 5. **Dirty data handling**: For production environments, it is recommended to set `errorLimit.record` to 0 to ensure data quality 6. **DI nodes have no datasource field**: Unlike ODPS_SQL and similar nodes, the DI node's spec.json does not require a `datasource` field; datasource information is configured in `parameter.datasource` within the code JSON FILE:references/flowspec-guide.md # FlowSpec Format Reference FlowSpec is the standardized JSON description format for DataWorks data development nodes and workflows. This document provides detailed descriptions of each field's meaning, type, and constraints. ## Top-Level Structure Each FlowSpec file contains the following top-level fields: ```json { "version": "2.0.0", "kind": "Node", "spec": {} } ``` | Field | Type | Required | Description | |------|------|------|------| | `version` | string | Yes | FlowSpec version number, format `2.x.x`, currently recommended `"2.0.0"` | | `kind` | string | Yes | Resource type, see SpecKind enum below | **SpecKind enum values**: | kind | Description | |------|------| | `Node` | Single node | | `ManualNode` | Manual single node | | `CycleWorkflow` | Cycle-scheduled workflow | | `ManualWorkflow` | Manually triggered workflow | | `TriggerWorkflow` | Event-triggered workflow | | `TemporaryWorkflow` | Temporary workflow | | `Workflow` | Generic workflow | | `PaiFlow` | PAI workflow | | `Component` | Component | | `Resource` | Resource | | `Function` | Function | | `Table` | Table | | `BatchDeployment` | Batch deployment | | `DataSource` | Data source | | `DataQuality` | Data quality | | `DataService` | Data service | | `DataCatalog` | Data catalog | | `DataIntegrationJob` | Data integration job | | Field | Type | Required | Description | |------|------|------|------| | `metadata` | object | No | Custom metadata, not used in business logic | | `spec` | object | Yes | Resource definition body, structure varies by `kind` | --- ## Node Type Details When `kind` is `"Node"`, `spec` contains the following structure: ```json { "spec": { "nodes": [ ... ], "dependencies": [ ... ] } } ``` ### spec.nodes (Node Definition Array) `nodes` is an array that typically contains one node object. The complete fields of a node object are as follows: #### name (Node Name) | Property | Value | |------|------| | Type | string | | Required | Yes | | Constraints | Minimum 1 character, recommend using English letters, numbers, and underscores | Node name, must be unique within the same project. The name is also used for references in `outputs.nodeOutputs` and `dependencies`. ```json "name": "etl_daily_report" ``` #### id (Node Identifier) | Property | Value | |------|------| | Type | string | | Required | Yes (must be set equal to `name`) | | Constraints | Must exactly match the `name` field value | Node identifier used for matching `spec.dependencies[*].nodeId`. **Always set `id` equal to `name`** when creating nodes. Without an explicit `id`, the `CreateNode` API may silently drop `spec.dependencies`. ```json "name": "etl_daily_report", "id": "etl_daily_report" ``` #### recurrence (Scheduling Type) | Property | Value | |------|------| | Type | string | | Required | No | | Default | `"Normal"` | | Options | `"Normal"` / `"Pause"` / `"Skip"` | - **Normal**: Normal scheduling, runs automatically according to the trigger definition - **Pause**: Paused scheduling, the node will not execute automatically, but downstream nodes still trigger normally - **Skip**: Skip scheduling, the node is marked as successful but does not actually execute ```json "recurrence": "Normal" ``` #### priority | Property | Value | |------|------| | Type | integer | | Required | No | | Range | 1 ~ 8 | | Default | 1 | Higher values mean higher priority. When resources are insufficient, higher-priority nodes execute first. ```json "priority": 3 ``` #### script (Script Definition) script is the core configuration of a node, defining the code to execute and the runtime environment. ```json "script": { "language": "odps-sql", "runtime": { "command": "ODPS_SQL" }, "content": "SELECT * FROM my_table;", "path": "business_flow/data_processing/etl_daily", "parameters": [...] } ``` | Sub-field | Type | Required | Description | |--------|------|------|------| | `language` | string | No | Script language identifier, must match the `language` in the registry | | `runtime.command` | string | Yes | Node type identifier (e.g., `ODPS_SQL`, `DIDE_SHELL`), determines the runtime environment | | `runtime.engine` | string | No | Runtime engine identifier (e.g., specific engine version) | | `runtime.flinkConf` | object | No | Flink job configuration (for Flink-type nodes) | | `runtime.emrJobConfig` | object | No | EMR job configuration (for EMR-type nodes) | | `runtime.sparkConf` | object | No | Spark configuration (for Spark-type nodes) | | `content` | string | No | Script code content. Usually empty during local development (code is in a separate file); must be populated when submitting to the API | | `path` | string | No | Script file path (inherited from SpecFile parent class), the node's script path in DataWorks | | `extension` | string | No | Script file extension (e.g., `.sql`, `.sh`, `.py`) | | `parameters` | array | No | Scheduling parameter list | `language` and `runtime.command` must match. See the "Common Node Types" table in SKILL.md for common types. ##### parameters (Scheduling Parameters) `parameters` is an array where each element defines a parameter: ```json "parameters": [ { "name": "bizdate", "scope": "NodeParameter", "type": "System", "value": "$yyyymmdd" }, { "name": "hour", "scope": "NodeParameter", "type": "System", "value": "$[hh24]" }, { "name": "env", "scope": "NodeParameter", "type": "Constant", "value": "production" } ] ``` | Sub-field | Type | Required | Description | |--------|------|------|------| | `name` | string | Yes | Parameter name | | `scope` | string | No | Scope: `"NodeParameter"` (node level) or `"WorkflowParameter"` (workflow level) | | `type` | string | No | Parameter type: `"System"` (system variable), `"Constant"` (constant), `"NodeOutput"` (upstream output) | | `value` | string | Yes | Parameter value or system variable expression | **Common system variables**: | Variable Expression | Description | Example Value | |-----------|------|--------| | `$yyyymmdd` | Business date (T-1), format yyyyMMdd | `20260321` | | `$bizdate` | Same as `$yyyymmdd` | `20260321` | | `$yyyy` | Year of the business date | `2026` | | `$mm` | Month of the business date | `03` | | `$dd` | Day of the business date | `21` | | `$[yyyymmdd]` | Run date (T+0), format yyyyMMdd | `20260322` | | `$[yyyy-mm-dd]` | Run date, format yyyy-MM-dd | `2026-03-22` | | `$[hh24]` | Hour of the run time (24-hour format) | `14` | | `$[hh24miss]` | Run time, format HHmmss | `143000` | | `$[yyyymmdd-1]` | Run date minus 1 day | `20260321` | | `$[yyyymmdd+7]` | Run date plus 7 days | `20260329` | | `$[yyyymm-1]` | Run date minus 1 month | `202602` | | `$gmtdate` | Current timestamp | `20260322143000` | | `${out_table_name}` | Custom parameter reference | User-assigned value | **How to reference parameters in code**: - Shell scripts: Use `$bizdate` or `${bizdate}` directly - SQL scripts: Use `${bizdate}` format - Python scripts: Access via `sys.argv` or `os.environ` #### trigger (Scheduling Trigger) Defines the node's scheduling trigger method and timing. | Sub-field | Type | Required | Description | |--------|------|------|------| | `type` | string | Yes | `Scheduler` (scheduled) / `Manual` / `Streaming` / `Custom` / `None` | | `cron` | string | Conditionally required | 6-field cron expression (`second minute hour day month weekday`), required when type is `Scheduler` | | `startTime` | string | No | Scheduling start time, format `yyyy-MM-dd HH:mm:ss` | | `endTime` | string | No | Scheduling end time, format `yyyy-MM-dd HH:mm:ss` | | `timezone` | string | No | Timezone, default `"Asia/Shanghai"` | | `delaySeconds` | integer | No | Delay execution in seconds | | `calendarId` | string | No | Custom calendar ID | | `identifier` | string | No | Custom trigger identifier, used when type is `Custom` | | `cycleType` | string | No | `"Daily"` (daily scheduling) / `"NotDaily"` (hourly/minute-level) | For detailed cron expression configuration and scheduling cycle type descriptions, see [scheduling-guide.md](scheduling-guide.md). #### runtimeResource (Runtime Resource) Specifies the resource group for node execution. ```json "runtimeResource": { "resourceGroup": "${spec.runtimeResource.resourceGroup}" } ``` | Sub-field | Type | Required | Description | |--------|------|------|------| | `resourceGroup` | string | Yes | Resource group identifier (e.g., `S_res_group_xxx`). Recommend using the `${spec.runtimeResource.resourceGroup}` placeholder, with the actual value configured in `dataworks.properties` | | `resourceGroupId` | string | No | Resource group ID, optional | The resource group identifier can be obtained via the `ListResourceGroups` API. #### datasource (Data Source) Node types that require a data source must configure this field. Whether a data source is needed is determined by the `datasourceType` in the registry. ```json "datasource": { "name": "${spec.datasource.name}", "type": "odps" } ``` | Sub-field | Type | Required | Description | |--------|------|------|------| | `name` | string | Yes | Datasource name. Recommend using the `${spec.datasource.name}` placeholder | | `type` | string | Yes | Datasource type (e.g., `odps`, `hologres`, `flink`, `emr`, `clickhouse`). Must match the `datasourceType` for the command in the registry | For whether each node type requires a data source and the corresponding `datasourceType`, see the "Common Node Types" table in SKILL.md. #### outputs (Output Definition) Defines the node's output identifier, used for downstream dependency. ```json "outputs": { "nodeOutputs": [ { "data": "${projectIdentifier}.etl_daily_report" } ] } ``` | Sub-field | Type | Required | Description | |--------|------|------|------| | `nodeOutputs` | array | No | Node output array | | `nodeOutputs[].data` | string | Yes | Output identifier, format `projectIdentifier.nodeName`. **Must be globally unique within the project** — duplicate output names cause deployment failure | The output identifier is used for `depends[].output` references in `dependencies` of downstream nodes. `${projectIdentifier}` is defined in `dataworks.properties`. The downstream's `depends[].output` must be **character-for-character identical** to this value. #### inputs (Input Definition) Defines the node's input data. **Do not use `inputs.nodeOutputs` to configure dependencies**; dependencies are maintained via the `spec.dependencies` array only. In `spec.dependencies`, `nodeId` is a **self-reference** (current node's own name), and `depends[].output` is the upstream node's output. ```json "inputs": { "nodeOutputs": [ { "data": "${projectIdentifier}.upstream_node" } ] } ``` > **Note**: This field is NOT recommended for dependency configuration; use `spec.dependencies` instead. Do NOT dual-write both `inputs.nodeOutputs` and `spec.dependencies`. #### rerunMode / rerunTimes / rerunInterval (Rerun Configuration) | Field | Type | Default | Description | |------|------|--------|------| | `rerunMode` | string | `"Allowed"` | Rerun mode: `"Allowed"`, `"Denied"`, `"FailureAllowed"` (only allowed on failure) | | `rerunTimes` | integer | `0` | Auto-retry count (0 means no auto-retry) | | `rerunInterval` | integer | `180000` | Retry interval (milliseconds), default 180000 (3 minutes) | ```json "rerunMode": "FailureAllowed", "rerunTimes": 3, "rerunInterval": 60000 ``` #### timeout / timeoutUnit (Timeout Configuration) | Field | Type | Default | Description | |------|------|--------|------| | `timeout` | number | `4` | Timeout value | | `timeoutUnit` | string | `"HOURS"` | Timeout unit: `"SECONDS"` / `"MINUTES"` / `"HOURS"` | ```json "timeout": 4, "timeoutUnit": "HOURS" ``` The node will be automatically terminated after timeout. Adjust based on actual task duration. #### instanceMode (Instance Generation Mode) | Property | Value | |------|------| | Type | string | | Options | `"T+1"` / `"Immediately"` | | Default | `"T+1"` | - **T+1**: Instances are generated the next day. A scheduling node configured today will start generating instances tomorrow - **Immediately**: Instances are generated immediately. Instances can be generated and run on the same day as configuration ```json "instanceMode": "T+1" ``` #### Other Node Fields The following are other available fields in the Node model: | Field | Type | Description | |------|------|------| | `autoParse` | boolean | Whether to automatically parse dependencies (extract input/output tables from SQL scripts, etc.) | | `ignoreBranchConditionSkip` | boolean | Whether to ignore branch condition skip | | `description` | string | Node description | | `strategy` | object | Node runtime strategy configuration | | `fileResources` | array | List of file resources associated with the node | | `functions` | array | List of functions associated with the node | | `datasets` | array | List of datasets associated with the node | | `reference` | object | Node reference configuration | | `combined` | object | Combined node configuration | | `paramHub` | object | Parameter hub configuration, used for parameter passing | | `subflow` | object | Sub-workflow configuration, used for embedded workflows | | `paiflow` | object | PAI workflow configuration | | `dqcRule` | object | Data quality rule configuration | --- #### Control Nodes The following control nodes have separate detailed documentation; see `references/nodetypes/controller/`: | Node Type | Command | Description | Documentation | |----------|------|------|------| | Branch node | `CONTROLLER_BRANCH` | Routes to different downstream branches based on conditions | [CONTROLLER_BRANCH.md](../nodetypes/controller/CONTROLLER_BRANCH.md) | | Join node | `CONTROLLER_JOIN` | Merges multiple branches | [CONTROLLER_JOIN.md](../nodetypes/controller/CONTROLLER_JOIN.md) | | Assignment node | `CONTROLLER_ASSIGNMENT` | Passes script results to downstream | [CONTROLLER_ASSIGNMENT.md](../nodetypes/controller/CONTROLLER_ASSIGNMENT.md) | | Traverse node | `CONTROLLER_TRAVERSE` | for-each loop | [CONTROLLER_TRAVERSE.md](../nodetypes/controller/CONTROLLER_TRAVERSE.md) | | Cycle node | `CONTROLLER_CYCLE` | do-while loop | [CONTROLLER_CYCLE.md](../nodetypes/controller/CONTROLLER_CYCLE.md) | --- ### spec.dependencies (Dependencies) `dependencies` defines the dependency relationships between nodes. | Sub-field | Type | Required | Description | |--------|------|------|------| | `nodeId` | string | Yes | **Self-reference**: the current node's own `name` (the node that HAS this dependency, NOT the upstream node) | | `depends` | array | No | Dependency list | | `depends[].type` | string | Yes | `Normal` / `CrossCycleDependsOnSelf` / `CrossCycleDependsOnChildren` / `CrossCycleDependsOnOtherNode` | | `depends[].output` | string | Yes | **Upstream** node's output identifier (format: `projectIdentifier.upstreamNodeName`, root node is `projectIdentifier_root`). Must be character-for-character identical to the upstream's `outputs.nodeOutputs[].data` | | `depends[].sourceType` | string | No | `"System"` (auto-parsed) / `"Manual"` (manually configured) | | `variableDepends` | array | No | Variable-level dependencies | For complete dependency configuration details (usage of `spec.dependencies`, cross-workflow/cross-project/cross-cycle dependencies), see [workflow-guide.md](workflow-guide.md). --- ## Workflow Type When `kind` is `CycleWorkflow` (cycle-scheduled) or `ManualWorkflow` (manually triggered), the `spec.workflows` array defines the workflow: | Sub-field | Type | Required | Description | |--------|------|------|------| | `name` | string | Yes | Workflow name | | `script.path` | string | No | Workflow path | | `script.runtime.command` | string | Yes | Fixed as `"WORKFLOW"` | | `trigger` | object | No | Scheduling trigger (required for `CycleWorkflow`, not for `ManualWorkflow`) | | `strategy` | object | No | Workflow-level strategy (priority, timeout, rerunMode, failureStrategy, etc.) | For the complete workflow development process (creation, node orchestration, dependency configuration, deployment), see [workflow-guide.md](workflow-guide.md). --- ## Placeholder Mechanism FlowSpec supports two types of placeholders that must be replaced with actual values before API submission (values come from `dataworks.properties`): ### spec Placeholders Format: `${spec.xxx}` Used for values in spec.json that need to be replaced per environment: ```json "datasource": { "name": "${spec.datasource.name}", "type": "odps" } ``` Corresponding `dataworks.properties`: ```properties spec.datasource.name=my_odps_datasource ``` ### script Placeholders Format: `${script.xxx}` Used for values in code files that need to be replaced per environment: ```sql -- In the code file SELECT * FROM ${script.database}.my_table WHERE dt='${script.bizdate}'; ``` Corresponding `dataworks.properties`: ```properties script.database=my_db script.bizdate=20260101 ``` ### projectIdentifier Placeholder Format: `${projectIdentifier}` A special placeholder used in `outputs` and `dependencies` to reference the project identifier: ```json "outputs": { "nodeOutputs": [{ "data": "${projectIdentifier}.my_node" }] } ``` Corresponding `dataworks.properties`: ```properties projectIdentifier=my_project_name ``` --- ## Complete Examples For complete FlowSpec examples of nodes and workflows, see `assets/templates/`. FILE:references/nodetypes/adb_spark/ADB_SPARK.md # ADB Spark (ADB_SPARK) ## Overview - Compute engine: `ADB_SPARK` - Content format: json - Extension: `.adb.spark.json` - Data source type: `adb_spark` - Description: AnalyticDB Spark job configuration The ADB Spark node is used to develop and schedule AnalyticDB Spark tasks in DataWorks. It supports multi-language development with Java, Scala, and Python, and is suitable for large-scale data processing, real-time data analysis, complex queries, and machine learning scenarios. Through this node, you can incorporate Spark jobs into the DataWorks periodic scheduling system and orchestrate them with other types of data development tasks. ## Content Structure The node content is a JSON configuration for Spark jobs; the server-side will clear custom fields. Depending on the language type, the main configuration parameters are as follows: **Java / Scala Jobs:** | Parameter | Description | |-----------|-------------| | Main Jar Resource | Storage path of the JAR package on OSS | | Main Class | Main class of the task in the JAR package, e.g., `org.apache.spark.examples.SparkPi` | | Parameters | Parameters passed to the code, supports scheduling parameters `${var}` | | Configuration | Spark runtime parameters, e.g., `spark.driver.resourceSpec:medium` | **Python Jobs:** | Parameter | Description | |-----------|-------------| | Main Program Package | Storage path of the Python script on OSS | | Parameters | Parameters passed to the code, e.g., data file paths | | Configuration | Spark runtime parameter configuration | ## Prerequisites - An AnalyticDB for MySQL cluster (Enterprise Edition, Lakehouse Edition, or Basic Edition) has been created in the same region as the DataWorks workspace, with a Job-type resource group configured. - The DataWorks workspace has enabled the new Data Studio (Data Development). - The DataWorks resource group and the AnalyticDB cluster are in the same VPC, and the IP whitelist has been configured. - The AnalyticDB cluster has been added as a compute resource (type: AnalyticDB for Spark) to the workspace and has passed the resource group connectivity test. - If using OSS to store JAR packages or Python files, ensure OSS and the cluster are in the same region. ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_adb_spark", "script": { "path": "example_adb_spark", "runtime": { "command": "ADB Spark" }, "content": "{}" } } ] } } ``` ## Restrictions - Only workspaces with the new Data Studio (Data Development) enabled are supported. - The AnalyticDB cluster must have a Job-type resource group configured; otherwise, Spark jobs cannot be submitted. - Debug runs require configuring compute resources, resource groups, and compute CU runtime properties. ## Reference - [ADB Spark Node - Alibaba Cloud Documentation](https://help.aliyun.com/zh/dataworks/user-guide/adb-spark-node) FILE:references/nodetypes/adb_spark/ADB_SPARK_SQL.md # ADB Spark SQL (ADB_SPARK_SQL) ## Overview - Compute engine: `ADB_SPARK` - Content format: sql - Extension: `.adb.spark.sql` - Data source type: `adb_spark` - Description: AnalyticDB Spark SQL query task The ADB Spark SQL node is used to develop and schedule AnalyticDB Spark SQL tasks in DataWorks. Through this node, you can directly write SQL statements to query and analyze data in AnalyticDB, and incorporate them into the DataWorks periodic scheduling system for orchestration with other types of data development tasks. This node is suitable for data analysis and computation, periodic batch data processing, and cross-system data integration scenarios. ## Configuration The node requires the following runtime properties to be configured: | Parameter | Description | |-----------|-------------| | Compute Resource | Select the bound AnalyticDB for Spark compute resource | | ADB Compute Resource Group | Interactive-type resource group (Spark engine) configured in the AnalyticDB cluster | | Resource Group | DataWorks resource group that has passed the connectivity test | | Compute CU | Use the default value; generally no modification needed | The SQL editing area supports using `${variable_name}` syntax to define dynamic parameters (e.g., `$[yyyymmdd]` for date processing), external library references, internal table creation, OSS storage integration, and Parquet compression format settings. ## Prerequisites - An AnalyticDB for MySQL cluster (Enterprise Edition, Lakehouse Edition, or Basic Edition) has been created in the same region as the DataWorks workspace, with an Interactive-type resource group (Spark engine) configured. - The DataWorks workspace has enabled the new Data Studio (Data Development). - The DataWorks resource group and the AnalyticDB cluster are in the same VPC, and the IP whitelist has been configured. - The AnalyticDB cluster has been added as a compute resource (type: AnalyticDB for Spark) to the workspace and has passed the resource group connectivity test. - If using OSS storage, ensure OSS and the cluster are in the same region. ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_adb_spark_sql", "script": { "path": "example_adb_spark_sql", "runtime": { "command": "ADB Spark SQL" }, "content": "SELECT 1;" } } ] } } ``` ## Restrictions - Only workspaces with the new Data Studio (Data Development) enabled are supported. - The AnalyticDB cluster must have an Interactive-type resource group (Spark engine) configured; otherwise, Spark SQL tasks cannot be executed. - Debug runs require first configuring compute resources, ADB compute resource group, resource group, and compute CU runtime properties. ## Reference - [ADB Spark SQL Node - Alibaba Cloud Documentation](https://help.aliyun.com/zh/dataworks/user-guide/adb-spark-sql-node) FILE:references/nodetypes/ai/ALINK.md # Alink (ALINK) ## Overview - Compute engine: `ALGORITHM` - Content format: empty (no code) - Extension: `.alink.py` - Data source type: `pai` - Description: Flink-based machine learning algorithm node, runs on the PAI platform Alink is Alibaba's open-source machine learning algorithm platform, built on Apache Flink, supporting both batch and streaming computation modes. In DataWorks, the Alink node is used to schedule Alink algorithm tasks on the PAI platform, enabling automated scheduling and orchestration of common machine learning scenarios such as classification, regression, clustering, recommendation, and feature engineering. The content format of this node is empty, meaning the node itself does not directly contain algorithm code. Instead, it executes by associating with a pre-configured Alink algorithm task on the PAI platform. Users must first build and debug the algorithm workflow on the PAI platform, then use the DataWorks Alink node for periodic scheduling. ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_alink", "script": { "path": "example_alink", "runtime": { "command": "alink" }, "content": "" } } ] } } ``` ## Restrictions - PAI (Machine Learning Platform) must be activated and bound to the DataWorks workspace. - Algorithm tasks must be created and configured on the PAI platform first; the Alink node itself does not support directly writing algorithm code (content format is empty). - Requires a `pai` type data source; the corresponding PAI data source connection must be configured in the workspace. FILE:references/nodetypes/ai/PAI.md # PAI (PAI) ## Overview - Compute engine: `ALGORITHM` - Content format: json - Extension: `.json` - Data source type: `pai` - Description: PAI command-line algorithm task node, scheduling PAI Command via JSON configuration PAI (Platform for AI) is Alibaba Cloud's artificial intelligence platform, providing algorithm capabilities covering the full lifecycle of machine learning and deep learning. In DataWorks, the PAI node is used to schedule PAI command-line tasks (PAI Command), specifying the algorithm name, project, and input/output table parameters via JSON-formatted content configuration, enabling periodic scheduling and automated orchestration of machine learning tasks. The typical format for PAI Command is: `PAI -name -project -DinputTableName=xxx -DoutputTableName=xxx`. The node's `content` field carries these command configuration details in JSON format, which are parsed and executed by the PAI platform. ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_pai", "script": { "path": "example_pai", "runtime": { "command": "pai" }, "content": "{}" } } ] } } ``` ## Restrictions - PAI (Machine Learning Platform) must be activated and bound to the DataWorks workspace. - A `pai` type data source connection must be configured in the workspace to ensure DataWorks can access PAI platform resources. - The `content` field is a JSON-formatted string that must contain valid PAI Command configuration; an empty object `{}` is only for placeholder purposes and must be filled with complete algorithm parameters for actual execution. - Input and output tables referenced in PAI Command must already exist in the corresponding MaxCompute project, and the executing account must have the appropriate read/write permissions. FILE:references/nodetypes/ai/PAI_DLC.md # PAI DLC (PAI_DLC) ## Overview - Code: `1119` - Compute engine: `ALGORITHM` - Content format: empty (no code) - Extension: `.pai.dlc.sh` - Data source type: `pai` - Description: Integrates Alibaba Cloud PAI platform's DLC (Deep Learning Containers) distributed training service The PAI DLC node is used to schedule and run containerized deep learning training tasks from the PAI platform in DataWorks. DLC provides a containerized distributed training environment supporting mainstream deep learning frameworks such as TensorFlow and PyTorch. Users can load existing DLC tasks into DataWorks or directly write task code to enable periodic scheduling of training tasks. Documentation: ## Configuration ### Development Methods The PAI DLC node supports two development methods: 1. **Load Existing Task** -- Search by name and load a DLC task already created on the PAI platform; the system automatically generates the corresponding node code. 2. **Write Code Directly** -- Write DLC task code directly in the editor, supporting dynamic scheduling parameters using `${variable_name}`. ### Common Parameters The following key parameters can be configured in the code: | Parameter | Description | |-----------|-------------| | `--name` | Task name | | `--command` | Execution command | | `--workspace_id` | PAI workspace ID | | `--priority` | Task priority (1-9) | | `--workers` | Number of compute nodes | | `--worker_spec` | Compute node specification | ### Workflow 1. Write or load DLC task code 2. Configure the scheduling resource group and run tests 3. Configure the scheduling period and dependency relationships 4. Publish the node to the production environment 5. Monitor run status in the Operations Center ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_pai_dlc", "script": { "path": "example_pai_dlc", "runtime": { "command": "pai_dlc" }, "content": "" } } ] } } ``` ## Restrictions - DataWorks must be authorized to access the PAI service before use (one-click authorization supported). - The node content format is empty; task code is managed on the PAI platform side, and DataWorks does not directly edit script content. - The PAI workspace and DataWorks workspace must be in the same region. FILE:references/nodetypes/ai/PAI_FLOW.md # PAI Flow (PAI_FLOW) ## Overview - Compute engine: `ALGORITHM` - Content format: yaml - Extension: `.yaml` - Data source type: `pai` - Description: PAI visual modeling workflow node for end-to-end machine learning pipeline development PAI Flow provides end-to-end machine learning pipeline development capabilities, sharing the same workflow functionality as the visual modeling designer on Alibaba Cloud's PAI platform. Users can drag and drop nodes such as data reading and RAG data processing onto the canvas and manually connect them to form workflows. PAI Flow nodes support periodic scheduling, enabling automated execution of AI tasks in DataWorks. Documentation: https://help.aliyun.com/zh/dataworks/user-guide/pai-flow-node ## Configuration ### Supported Sub-node Types **Source/Target Nodes:** - Read Data Table -- Read MaxCompute table data - Read OSS Data -- Read files or folders from object storage paths - Read CSV File -- Supports reading CSV format data from OSS, HTTP, HDFS - Write Data Table -- Write data to MaxCompute **RAG Data Processing Nodes:** - RAG Text Parse and Chunk -- Parse text files and generate text chunks of specified sizes - RAG Vector Generation -- Generate text vectors using Embedding models - RAG Knowledge Base Index Sync -- Sync data to target knowledge base indexes ### Variable Support File paths support configuring variables (e.g., `${variable}/example.csv`), which can be combined with scheduling parameters for dynamic paths during periodic scheduling. ### Prerequisites - When creating a DataWorks workspace, check "Create AI workspace with same name"; for existing workspaces, enable "Schedule PAI algorithm tasks" in the Management Center - Only Serverless resource groups are supported ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_pai_flow", "script": { "path": "example_pai_flow", "runtime": { "command": "PAI_FLOW" }, "content": "" } } ] } } ``` ## Restrictions - Only supported in new DataWorks workspaces - Currently only supports source/target nodes and RAG data processing nodes - Only Serverless resource groups are supported for execution - Region restrictions: China East 1 (Hangzhou), China East 2 (Shanghai), China North 2 (Beijing), China North 6 (Ulanqab), China South 1 (Shenzhen), Hong Kong, Singapore, Jakarta, Tokyo, Frankfurt, Silicon Valley, Virginia FILE:references/nodetypes/ai/PAI_STUDIO.md # PAI Studio (PAI_STUDIO) ## Overview - Code: `1117` - Compute engine: `ALGORITHM` - Content format: json - Extension: `.json` - Data source type: `pai` - Description: PAI visual modeling experiment node for building machine learning workflows via drag-and-drop and scheduling them in DataWorks PAI Studio (now known as PAI Designer) is the visual modeling tool provided by Alibaba Cloud's PAI platform. In DataWorks, the PAI Studio node is used to load and schedule machine learning experiment workflows created on PAI Designer, enabling periodic automated execution of end-to-end machine learning development processes. Users can arrange components for data preprocessing, feature engineering, model training, and model evaluation by dragging and dropping on PAI Designer, then configure scheduling parameters and dependencies in DataWorks to incorporate the experiment workflow into the production scheduling system. Documentation: ## Configuration ### Development Methods The PAI Studio node supports the following methods for creating workflows: 1. **Create Blank Workflow** -- Start from scratch on the PAI Designer canvas, dragging algorithm components and connecting them to build experiments. 2. **Create from Preset Template** -- Use platform-provided templates for quick start, suitable for common machine learning scenarios. 3. **Use Custom Template** -- Create workflows based on team-customized templates for team collaboration and reuse. ### Scheduling Parameters Supports defining dynamic parameters using `${variable_name}` syntax in workflows, assigning values in the DataWorks scheduling configuration for dynamic input across different scheduling cycles. ### Workflow 1. Create a PAI Designer node in DataWorks 2. Enter PAI Designer to create or edit a machine learning workflow 3. Configure scheduling parameters (if dynamic input is needed) 4. Save and publish the node to the production environment 5. Execute tasks in the Operations Center via "Test" or "Backfill" > Note: PAI Studio nodes do not have a direct run entry; they must be triggered via the Operations Center. ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_pai_studio", "script": { "path": "example_pai_studio", "runtime": { "command": "pai_studio" }, "content": "{}" } } ] } } ``` ## Restrictions - DataWorks must be authorized to access PAI services before use; only the primary account or RAM users with `AliyunDataWorksFullAccess` permission can perform the authorization. - A `pai` type data source connection must be configured in the workspace to ensure DataWorks can access PAI platform resources. - The `content` field is a JSON-formatted string carrying workflow configuration information; an empty object `{}` is for placeholder only and must contain a valid experiment workflow configuration for actual execution. - The node does not support direct execution in the DataWorks editor; it must be published and then triggered via test or backfill in the Operations Center. FILE:references/nodetypes/ai/RECOMMEND_PLUS.md # Recommendation Engine (RECOMMEND_PLUS) ## Overview - Compute engine: `ALGORITHM` - Content format: json - Extension: `.json` - Data source type: `pai` - Description: Scheduling node for Alibaba Cloud Intelligent Recommendation (AIRec) in DataWorks, for periodically updating recommendation data The RECOMMEND_PLUS node incorporates Alibaba Cloud Intelligent Recommendation Engine data processing tasks into the DataWorks scheduling system. Through this node, users can perform unified periodic scheduling management of tasks such as recommendation engine data backflow, feature processing, and model training, ensuring that recommendation data updates are coordinated with upstream and downstream data processing workflows. ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_recommend_plus", "script": { "path": "example_recommend_plus", "runtime": { "command": "RECOMMEND_PLUS" }, "content": "{}" } } ] } } ``` ## Restrictions - Alibaba Cloud Intelligent Recommendation (AIRec) service must be activated first, and the corresponding PAI data source must be bound in the DataWorks workspace - Node content is in JSON format; the specific configuration structure is defined by the Intelligent Recommendation Engine FILE:references/nodetypes/ai/XLAB.md # XLab (XLAB) ## Overview - Code: `87` - Compute engine: `ALGORITHM` - Content format: empty (no code) - Extension: `.xlab.json` - Data source type: `pai` - Description: Early visual data exploration and analysis node on the PAI platform, for conducting data analysis experiments via a graphical interface XLab is an early visual data exploration and analysis tool provided by Alibaba Cloud's PAI platform. In DataWorks, the XLab node is used to schedule data analysis experiments created on the XLab platform, supporting graphical exploratory analysis, statistical descriptions, and visual presentations of data, and incorporating analysis workflows into the DataWorks scheduling system for periodic execution. This node type belongs to an earlier generation of algorithm nodes and has been gradually replaced by PAI Designer (PAI_STUDIO) and other next-generation visual modeling tools. For new scenarios, PAI Designer nodes are recommended. ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_xlab", "script": { "path": "example_xlab", "runtime": { "command": "xlab" }, "content": "" } } ] } } ``` ## Restrictions - This node type is an early algorithm node that has been gradually replaced by PAI Designer (PAI_STUDIO); PAI Designer nodes are recommended for new projects. - PAI (Machine Learning Platform) must be activated and bound to the DataWorks workspace. - A `pai` type data source connection must be configured in the workspace to ensure DataWorks can access PAI platform resources. - The `content` field is an empty string (empty format); experiment configuration is completed through the XLab platform's visual interface rather than written directly in the node script. FILE:references/nodetypes/cdh/CDH_FILE.md # CDH File Resource (CDH_FILE) ## Overview - Compute engine: `HADOOP_CDH` - Content format: json - Extension: `.json` - Data source type: `cdh` - Label type: RESOURCE - Description: Manages general file resources on CDH clusters Used to register and manage general file resources (such as configuration files, data files, etc.) used by CDH clusters in DataWorks. Registered file resources can be referenced by CDH series compute nodes. ## Prerequisites - CDH cluster has been added as a compute resource to the DataWorks workspace - DataWorks resource group has network connectivity with the CDH cluster - The files to upload have been prepared ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_cdh_file", "script": { "path": "example_cdh_file", "runtime": { "command": "CDH_FILE" }, "content": "{}" } } ] } } ``` FILE:references/nodetypes/cdh/CDH_FUNCTION.md # CDH Function (CDH_FUNCTION) ## Overview - Compute engine: `HADOOP_CDH` - Content format: empty (no code) - Extension: none - Data source type: `cdh` - Label type: FUNCTION - Description: Manages custom functions on CDH clusters Used to register and manage custom functions (UDFs) on CDH clusters in DataWorks. Registered functions can be called in CDH Hive SQL, Spark SQL, and other nodes. ## Prerequisites - CDH cluster has been added as a compute resource to the DataWorks workspace - DataWorks resource group has network connectivity with the CDH cluster - JAR resources that the function implementation depends on have been uploaded ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_cdh_function", "script": { "path": "example_cdh_function", "runtime": { "command": "CDH_FUNCTION" }, "content": "" } } ] } } ``` ## Restrictions\n\n- The node content format is empty; function definitions specify the class name and associated JAR resources through metadata configuration FILE:references/nodetypes/cdh/CDH_HIVE.md # CDH Hive SQL (CDH_HIVE) ## Overview - Compute engine: `HADOOP_CDH` - Content format: sql - Extension: `.sql` - Data source type: `cdh` - Description: Executes Hive SQL statements on CDH clusters Used to run SQL queries and data processing tasks on the Hive engine of Cloudera CDH clusters. Suitable for Hive-based ETL data processing, table creation, and data query scenarios. ## Prerequisites - CDH cluster has been added as a compute resource to the DataWorks workspace - DataWorks resource group has network connectivity with the CDH cluster - CDH cluster has Hive service deployed ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_cdh_hive", "script": { "path": "example_cdh_hive", "runtime": { "command": "CDH_HIVE" }, "content": "SELECT 1;" } } ] } } ``` FILE:references/nodetypes/cdh/CDH_IMPALA.md # CDH Impala SQL (CDH_IMPALA) ## Overview - Compute engine: `HADOOP_CDH` - Content format: sql - Extension: `.sql` - Data source type: `cdh` - Description: Executes Impala SQL queries on CDH clusters Used to run SQL queries on the Impala engine of Cloudera CDH clusters. Impala provides low-latency interactive query capabilities for HDFS and HBase data, suitable for ad-hoc queries and BI analysis scenarios requiring fast response times. ## Prerequisites - CDH cluster has been added as a compute resource to the DataWorks workspace - DataWorks resource group has network connectivity with the CDH cluster - CDH cluster has Impala service deployed ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_cdh_impala", "script": { "path": "example_cdh_impala", "runtime": { "command": "CDH_IMPALA" }, "content": "SELECT 1;" } } ] } } ``` FILE:references/nodetypes/cdh/CDH_JAR.md # CDH JAR Resource (CDH_JAR) ## Overview - Compute engine: `HADOOP_CDH` - Content format: json - Extension: `.json` - Data source type: `cdh` - Label type: RESOURCE - Description: Manages JAR resource files on CDH clusters Used to register and manage JAR resources used by CDH clusters in DataWorks. Registered JAR resources can be referenced by CDH MapReduce, CDH Spark, and other nodes. ## Prerequisites - CDH cluster has been added as a compute resource to the DataWorks workspace - DataWorks resource group has network connectivity with the CDH cluster - The JAR files to upload have been prepared ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_cdh_jar", "script": { "path": "example_cdh_jar", "runtime": { "command": "CDH_JAR" }, "content": "{}" } } ] } } ``` FILE:references/nodetypes/cdh/CDH_MR.md # CDH MapReduce (CDH_MR) ## Overview - Compute engine: `HADOOP_CDH` - Content format: empty (no code) - Extension: none - Data source type: `cdh` - Description: Runs MapReduce jobs on CDH clusters Used to submit and run Hadoop MapReduce jobs on Cloudera CDH clusters. MapReduce jobs specify the JAR package and main class through configuration parameters; the node itself does not contain code content. ## Prerequisites - CDH cluster has been added as a compute resource to the DataWorks workspace - DataWorks resource group has network connectivity with the CDH cluster - CDH cluster has MapReduce/YARN service deployed - JAR resources needed for the MapReduce job have been uploaded ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_cdh_mr", "script": { "path": "example_cdh_mr", "runtime": { "command": "CDH_MR" }, "content": "" } } ] } } ``` ## Restrictions\n\n- The node content format is empty; job logic must be specified through associated JAR resources and runtime parameters FILE:references/nodetypes/cdh/CDH_PRESTO.md # CDH Presto SQL (CDH_PRESTO) ## Overview - Compute engine: `HADOOP_CDH` - Content format: sql - Extension: `.sql` - Data source type: `cdh` - Description: Executes Presto SQL queries on CDH clusters Used to run SQL queries on the Presto engine of Cloudera CDH clusters. Presto is a distributed SQL query engine suitable for interactive low-latency queries across multiple data sources. ## Prerequisites - CDH cluster has been added as a compute resource to the DataWorks workspace - DataWorks resource group has network connectivity with the CDH cluster - CDH cluster has Presto service deployed ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_cdh_presto", "script": { "path": "example_cdh_presto", "runtime": { "command": "CDH_PRESTO" }, "content": "SELECT 1;" } } ] } } ``` FILE:references/nodetypes/cdh/CDH_SHELL.md # CDH Shell (CDH_SHELL) ## Overview - Compute engine: `HADOOP_CDH` - Content format: shell - Extension: `.sh` - Data source type: `cdh` - Description: Executes Shell scripts on CDH clusters Used to execute Shell scripts on the gateway node of Cloudera CDH clusters. Can invoke various command-line tools on the CDH cluster (such as hdfs, hadoop, etc.), suitable for file operations, cluster management, and custom script tasks. ## Prerequisites - CDH cluster has been added as a compute resource to the DataWorks workspace - DataWorks resource group has network connectivity with the CDH cluster ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_cdh_shell", "script": { "path": "example_cdh_shell", "runtime": { "command": "CDH_SHELL" }, "content": "#!/bin/bash\necho 'hello'" } } ] } } ``` FILE:references/nodetypes/cdh/CDH_SPARK.md # CDH Spark Job (CDH_SPARK) ## Overview - Compute engine: `HADOOP_CDH` - Content format: shell - Extension: `.sh` - Data source type: `cdh` - Description: Submits Spark jobs on CDH clusters Used to submit and run Spark jobs on Cloudera CDH clusters via Shell scripts. The script can contain spark-submit commands to submit Spark JAR packages or Python scripts. ## Prerequisites - CDH cluster has been added as a compute resource to the DataWorks workspace - DataWorks resource group has network connectivity with the CDH cluster - CDH cluster has Spark service deployed ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_cdh_spark", "script": { "path": "example_cdh_spark", "runtime": { "command": "CDH_SPARK" }, "content": "#!/bin/bash\necho 'hello'" } } ] } } ``` FILE:references/nodetypes/cdh/CDH_SPARK_SHELL.md # CDH Spark Shell (CDH_SPARK_SHELL) ## Overview - Compute engine: `HADOOP_CDH` - Content format: shell - Extension: `.sh` - Data source type: `cdh` - Description: Runs Spark-related scripts in Shell mode on CDH clusters Used to execute scripts via Spark Shell mode on Cloudera CDH clusters. Similar to CDH_SPARK but focused on batch submission scenarios for interactive Spark scripts. ## Prerequisites - CDH cluster has been added as a compute resource to the DataWorks workspace - DataWorks resource group has network connectivity with the CDH cluster - CDH cluster has Spark service deployed ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_cdh_spark_shell", "script": { "path": "example_cdh_spark_shell", "runtime": { "command": "CDH_SPARK_SHELL" }, "content": "#!/bin/bash\necho 'hello'" } } ] } } ``` FILE:references/nodetypes/cdh/CDH_SPARK_SQL.md # CDH Spark SQL (CDH_SPARK_SQL) ## Overview - Compute engine: `HADOOP_CDH` - Content format: sql - Extension: `.sql` - Data source type: `cdh` - Description: Executes Spark SQL statements on CDH clusters Used to run SQL queries on the Spark SQL engine of Cloudera CDH clusters. Compared to Hive SQL, Spark SQL typically offers better execution performance, suitable for data processing scenarios requiring higher query efficiency. ## Prerequisites - CDH cluster has been added as a compute resource to the DataWorks workspace - DataWorks resource group has network connectivity with the CDH cluster - CDH cluster has Spark service deployed ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_cdh_spark_sql", "script": { "path": "example_cdh_spark_sql", "runtime": { "command": "CDH_SPARK_SQL" }, "content": "SELECT 1;" } } ] } } ``` FILE:references/nodetypes/cdh/CDH_TABLE.md # CDH Table (CDH_TABLE) ## Overview - Compute engine: `HADOOP_CDH` - Content format: empty (no code) - Extension: none - Data source type: `cdh` - Label type: TABLE - Description: Manages Hive table metadata on CDH clusters Used to manage table objects (such as Hive tables) on CDH clusters in DataWorks. This node type is for table metadata registration and management and does not contain executable code. ## Prerequisites - CDH cluster has been added as a compute resource to the DataWorks workspace - DataWorks resource group has network connectivity with the CDH cluster - CDH cluster has Hive Metastore service deployed ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_cdh_table", "script": { "path": "example_cdh_table", "runtime": { "command": "CDH_TABLE" }, "content": "" } } ] } } ``` ## Restrictions\n\n- The node content format is empty; table structure is defined through metadata configuration rather than code FILE:references/nodetypes/controller/CONTROLLER_ASSIGNMENT.md # Assignment Node（CONTROLLER_ASSIGNMENT） ## Overview The assignment node is a controller node that passes script execution results to downstream nodes.After execution, it automatically assigns the **last output** to `${outputs}` variable; downstream nodes reference it via `inputs.variables` . ``` Assignment Node A (executes script -> produces ${outputs}) │ ▼ Downstream Node B (references upstream outputs via ${param}) ``` **Key restriction**: Parameters can only be passed to **direct downstream (one level)** child nodes; cross-level passing is not supported. ## Supported Languages and Output Rules | Language | script.language | Output Rule | Transfer Format | |------|----------------|-------------|---------| | MaxCompute SQL | `odps` | The result of the last `SELECT` statement | 2D array `[["v1","v2"],["v3","v4"]]` | | Python 2 | `python` | The output of the last `print` statement | 1D array `["v1","v2","v3"]` | | Shell | `shell` | The output of the last `echo` statement | 1D array `["v1","v2","v3"]` | Code examples for each language: ```sql -- MaxCompute SQL: The result of the last SELECT statement is assigned to outputs select col1, col2 from my_table where dt = '${bizdate}'; ``` ```python # Python 2: The output of the last print statement is assigned to outputs print("value1,value2,value3") ``` ```bash # Shell: The output of the last echo statement is assigned to outputs echo "value1,value2,value3" ``` When output content contains commas, use `\,` to escape: `echo "Electronics,Clothing\, Shoes"` → `["Electronics", "Clothing, Shoes"]` ## Code Storage Format The assignment node's code is stored in JSON format (extension `.assign.json`), containing two fields: ```json {"language": "odps", "content": "select 1"} ``` In `script.content`, it can be plain text (e.g., `"select 1"`) or a complete JSON string (e.g., `"{\"language\":\"odps\",\"content\":\"select 1\"}"`); the system handles both formats automatically. ## Parameter Passing Configuration ### Assignment Node outputs.variables Declare the output variables of this node for downstream reference: ```json "outputs": { "variables": [ { "artifactType": "Variable", "name": "my_assign_node", "scope": "NodeContext", "type": "NodeOutput", "value": "${outputs}", "node": {"output": ""} } ], "outputs": [ { "artifactType": "NodeOutput", "data": "", "refTableName": "my_assign_node" } ] } ``` ### Downstream Node inputs.variables Reference the output of the upstream assignment node: ```json "inputs": { "variables": [ { "artifactType": "Variable", "name": "my_assign_node", "scope": "NodeContext", "type": "NodeOutput", "value": "${outputs}", "node": {"output": ""} } ] } ``` In the downstream node code, use `${my_assign_node}` to reference the passed value. ### Downstream Node script.parameters References must also be declared in parameters: ```json "parameters": [ { "artifactType": "Variable", "name": "my_assign_node", "scope": "NodeContext", "type": "NodeOutput", "value": "${outputs}", "node": {"output": ""} } ] ``` ## Full Example Assignment node (MaxCompute SQL, passing query results to downstream): ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "sql_assign_1", "script": { "path": "sql_assign_1", "language": "odps", "content": "select user_id, user_name from dim_user where status = 1;", "runtime": { "command": "CONTROLLER_ASSIGNMENT" } }, "outputs": { "variables": [ { "artifactType": "Variable", "name": "sql_assign_1", "scope": "NodeContext", "type": "NodeOutput", "value": "${outputs}", "node": {"output": "${projectIdentifier}.sql_assign_1"} } ], "nodeOutputs": [ { "data": "${projectIdentifier}.sql_assign_1", "artifactType": "NodeOutput" } ] } } ], "dependencies": [ { "nodeId": "sql_assign_1", "depends": [ {"type": "Normal", "output": "${projectIdentifier}_root"} ] } ] } } ``` Downstream node referencing assignment results: ```json { "name": "downstream_node", "script": { "path": "downstream_node", "runtime": {"command": "ODPS_SQL"}, "content": "select * from my_table where user_id in (${sql_assign_1});", "parameters": [ { "artifactType": "Variable", "name": "sql_assign_1", "scope": "NodeContext", "type": "NodeOutput", "value": "${outputs}", "node": {"output": "${projectIdentifier}.sql_assign_1"} } ] }, "inputs": { "variables": [ { "artifactType": "Variable", "name": "sql_assign_1", "scope": "NodeContext", "type": "NodeOutput", "value": "${outputs}", "node": {"output": "${projectIdentifier}.sql_assign_1"} } ], "nodeOutputs": [ { "data": "${projectIdentifier}.sql_assign_1", "artifactType": "NodeOutput" } ] } } ``` ## Restrictions | Restriction | Details | |--------|------| | Pass level | Can only pass to direct downstream (one level) child nodes | | Pass size | Maximum 2MB; exceeding this will cause execution failure | | Code comments | Comments are not supported | | SQL syntax | MaxCompute SQL does not support WITH syntax | | Python version | Only Python 2 is supported | | Comma escaping | Use `\,` to escape commas in output | ## Common Errors | Error | Cause | Solution | |------|------|------| | `find no select sql in sql assignment!` | SQL mode missing SELECT | Ensure code contains a SELECT query | | `OutPut Result is null, cannot handle!` | Python/Shell missing output statement | Add `print()` or `echo` | | Output array split unexpectedly | Comma not escaped | Use `\,` to escape | | Downstream node cannot obtain parameters | Cross-level reference or inputs.variables configuration error | Only reference direct upstream; check node.output | | Node execution failed without specific error | Output exceeds 2MB | Simplify query results | FILE:references/nodetypes/controller/CONTROLLER_BRANCH.md # Branch Node（CONTROLLER_BRANCH） ## Overview The branch node is a controller node that dispatches the workflow to different downstream branches based on condition expressions.Multiple branch conditions are defined via the `branch.branches` array; at runtime, they are matched in order, and the branch whose condition is met gets activated. ## Configuration Branch conditions are configured via the `branch` field: ```json { "name": "my_branch", "script": { "path": "my_branch", "runtime": { "command": "CONTROLLER_BRANCH" }, "content": "" }, "branch": { "branches": [ { "when": "${status} == 'success'", "output": "branch_success_output" }, { "when": "${status} == 'failure'", "output": "branch_failure_output" } ] } } ``` ### Field Description | Field | Type | Description | |------|------|------| | `branch.branches` | array | List of branch conditions, matched in order | | `branches[].when` | string | Branch condition expression | | `branches[].output` | string | Output identifier when the branch condition is met; downstream nodes establish dependencies via this identifier | ### script.content The branch node's `script.content` is an empty string `""`. ## Full Example ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "recurrence": "Normal", "name": "my_branch", "script": { "path": "my_branch", "runtime": { "command": "CONTROLLER_BRANCH" }, "content": "" }, "branch": { "branches": [ { "when": "${status} == 'success'", "output": "${projectIdentifier}.my_branch.branch_success" }, { "when": "${status} == 'failure'", "output": "${projectIdentifier}.my_branch.branch_failure" } ] }, "outputs": { "nodeOutputs": [ { "data": "${projectIdentifier}.my_branch", "artifactType": "NodeOutput" } ] } } ], "dependencies": [ { "nodeId": "my_branch", "depends": [ { "type": "Normal", "output": "${projectIdentifier}_root" } ] } ] } } ``` FILE:references/nodetypes/controller/CONTROLLER_CYCLE.md # Loop Node（CONTROLLER_CYCLE） ## Overview - Compute engine: `GENERAL` - Content format: json - Extension: `.do-while.json` The loop node implements do-while loop logic: execute the loop body first, then determine whether to continue. It can be used standalone or combined with assignment nodes to iterate over result sets. ## Node Structure ``` do-while wrapper (CONTROLLER_CYCLE) ├── Start Node (CONTROLLER_CYCLE_START, auto-generated, cannot be deleted) ├── Loop body business nodes (multiple can be added, custom arrangement) └── End Node (CONTROLLER_CYCLE_END, auto-generated, determines whether to continue looping) ``` - Start node: Marks the beginning of the loop; has no business function - Loop body: Multiple child nodes can be added and executed by dependency - End node: Executes evaluation code; returns True to continue, False to exit **Execution flow**: Start -> Execute loop body by dependencies -> End evaluates -> If True, return to Start; if False, exit ## Loop Condition The end node uses code to determine whether to continue looping: ```python # Control loop 5 times if ${dag.loopTimes} < 5: print True else: print False ``` - Returns `True`: Continue to the next loop iteration - Returns `False`: Exit the loop When iterating over a dataset with an assignment node: ```python # Iterate by dataset length if ${dag.loopTimes} <= ${dag.input.length}: print True else: print False ``` ## Built-in Variables | Variable | Description | Example | |------|------|------| | `${dag.loopTimes}` | Current loop count (starting from 1) | 1st time = 1, 2nd time = 2 | | `${dag.offset}` | Offset (starting from 0) | 1st time = 0, 2nd time = 1 | | `${dag.input}` | Dataset passed by assignment node | Array format | | `${dag.input[${dag.offset}]}` | Data row for the current loop | Indexed by offset | | `${dag.input.length}` | Dataset length | Total record count | ## Configuration ### script.content Field Stores loop control configuration in JSON format: | Field | Type | Default | Description | |------|------|--------|------| | `maxIterations` | number | 128 | Maximum iterations; exceeding this will cause an error | | `parallelism` | number | 0 | Concurrency not supported; must be serial (next cycle starts only after the previous one completes) | ### dowhile Field | Field | Type | Description | |------|------|------| | `dowhile.nodes` | array | List of child nodes within the loop body | | `dowhile.flow` | array | Dependency relationships of child nodes within the loop body | | `dowhile.while` | string/object | Loop continuation condition | ## Full Example ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "recurrence": "Normal", "name": "my_loop", "script": { "path": "my_loop", "runtime": { "command": "CONTROLLER_CYCLE" }, "content": "{\"maxIterations\":128,\"parallelism\":0}" }, "dowhile": { "nodes": [ { "name": "loop_task", "script": { "path": "my_loop/loop_task", "runtime": { "command": "DIDE_SHELL" }, "content": "echo \"loop iteration: ${dag.loopTimes}, offset: ${dag.offset}\"" } } ], "flow": [ { "nodeId": "loop_task", "depends": [] } ], "while": "${dag.loopTimes} < 5" }, "outputs": { "nodeOutputs": [ { "data": "${projectIdentifier}.my_loop", "artifactType": "NodeOutput" } ] } } ], "flow": [ { "nodeId": "my_loop", "depends": [ { "type": "Normal", "output": "${projectIdentifier}_root" } ] } ] } } ``` ## Restrictions | Restriction | Details | |--------|------| | Version requirement | DataWorks Standard Edition or above | | Maximum iterations | 128; exceeding this will cause an error | | Concurrency | Concurrent execution not supported; serial only | | Test method | Cannot be tested directly in Data Studio; must be published and executed in the Operations Center | | Internal branches | When using branch nodes, merge nodes must also be used | | End node code | Comments are not supported | | With assignment node | When backfilling data, both the assignment node and loop node must be selected; otherwise, the passed values cannot be retrieved | FILE:references/nodetypes/controller/CONTROLLER_CYCLE_END.md # Loop End Node（CONTROLLER_CYCLE_END） ## Overview - Compute engine: `GENERAL` - Content format: empty (no code) - Extension: `.do-while-end` - Description: do-while loop end (auto-generated) ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_controller_cycle_end", "script": { "path": "example_controller_cycle_end", "runtime": { "command": "CONTROLLER_CYCLE_END" }, "content": "" } } ] } } ``` FILE:references/nodetypes/controller/CONTROLLER_CYCLE_START.md # Loop Start Node（CONTROLLER_CYCLE_START） ## Overview - Compute engine: `GENERAL` - Content format: empty (no code) - Extension: `.do-while-start` - Description: do-while loop start (auto-generated) ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_controller_cycle_start", "script": { "path": "example_controller_cycle_start", "runtime": { "command": "CONTROLLER_CYCLE_START" }, "content": "" } } ] } } ``` FILE:references/nodetypes/controller/CONTROLLER_JOIN.md # Merge Node（CONTROLLER_JOIN） ## Overview - Compute engine: `GENERAL` - Content format: empty (no code) - Extension: `.join.json` The merge node is a logical control node used to consolidate the run status of upstream nodes, solving dependency mounting and run triggering issues downstream of branch nodes. **Typical scenario**: Branch node C defines two mutually exclusive branches C1 and C2 that write data to the same table. Downstream node B cannot directly depend on C1 and C2 (unselected branches will cause B to empty-run). A merge node J must first consolidate the two branches, then let B depend on J. ``` Branch Node C ├── C1 (executed when condition is met) └── C2 (executed when condition is not met) ↘ ↙ Merge Node J ↓ Downstream Node B ``` ## Merge Conditions | Logic | Description | |------|------| | AND | All branches must be in terminal state and all must satisfy the configured run status for the merge node to be marked as success | | OR | All branches must be in terminal state; if any branch satisfies the configured run status, the merge node is marked as success | **Node run status options**: - Success: Node ran successfully - Failure: Node ran and failed - Branch not run: Empty-run status when a node is not selected (only applies when the upstream is a branch node) ## Configuration Merging is achieved through dual-writing dependencies via `inputs.nodeOutputs` + `flow.depends`. ```json { "recurrence": "Normal", "name": "join_node", "script": { "path": "join_node", "runtime": { "command": "CONTROLLER_JOIN" }, "content": "" }, "inputs": { "nodeOutputs": [ { "data": "branch_a_output" }, { "data": "branch_b_output" } ] } } ``` Also declare dependencies in `flow.depends`: ```json { "nodeId": "join_node", "depends": [ { "nodeId": "branch_a", "type": "Normal" }, { "nodeId": "branch_b", "type": "Normal" } ] } ``` ### Field Description | Configuration | Description | |--------|------| | `script.content` | Must be an empty string `""`; cannot be `"{}"` | | `inputs.nodeOutputs` | List all upstream branch outputs that need to be merged | | `flow.depends` | Keep consistent with `inputs.nodeOutputs`, dual-write dependencies | ## Full Example Used with branch nodes to merge two branches: ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "recurrence": "Normal", "name": "join_node", "script": { "path": "join_node", "runtime": { "command": "CONTROLLER_JOIN" }, "content": "" }, "inputs": { "nodeOutputs": [ { "data": "${projectIdentifier}.branch_a", "artifactType": "NodeOutput" }, { "data": "${projectIdentifier}.branch_b", "artifactType": "NodeOutput" } ] }, "outputs": { "nodeOutputs": [ { "data": "${projectIdentifier}.join_node", "artifactType": "NodeOutput" } ] } } ], "flow": [ { "nodeId": "join_node", "depends": [ { "nodeId": "branch_a", "type": "Normal" }, { "nodeId": "branch_b", "type": "Normal" } ] } ] } } ``` ## Restrictions | Restriction | Details | |--------|------| | Version requirement | DataWorks Standard Edition or above | | Execution result | Currently only supports setting to success status | | Node type | Logical control node; does not perform computation | FILE:references/nodetypes/controller/CONTROLLER_TRAVERSE.md # Traverse Node（CONTROLLER_TRAVERSE） ## Overview - Compute engine: `GENERAL` - Content format: json - Extension: `.for-each.json` The traverse node implements for-each loop logic, automatically iterating over the result set output by the upstream assignment node and repeatedly executing the internal loop body for each element. **Use case**: Execute the same processing logic for multiple business units, product lines, or configuration items. ## Node Structure ``` Assignment Node (outputs array, e.g., "10,20,30") ↓ for-each wrapper (CONTROLLER_TRAVERSE) ├── Start Node (CONTROLLER_TRAVERSE_START, auto-generated) ├── Internal Business Node (actual processing logic) └── End Node (CONTROLLER_TRAVERSE_END, auto-generated) ``` - Wrapper container: Contains the entire loop workflow - Start/End nodes: Auto-generated, not editable, only mark loop body boundaries - Internal business nodes: Execute actual data processing ## Built-in Variables Internal nodes reference traverse data via the following variables: | Variable | Description | |------|------| | `${dag.loopDataArray}` | The complete result set from the upstream assignment node | | `${dag.foreach.current}` | The data item currently being processed in the loop | | `${dag.offset}` | Current loop offset (starting from 0) | | `${dag.loopTimes}` | Current loop iteration (starting from 1) | | `${dag.foreach.current[n]}` | The n-th data item in the current data row (2D array) | ## Configuration ### script.content Field Stores traverse control configuration in JSON format: | Field | Type | Default | Description | |------|------|--------|------| | `maxIterations` | number | 128 | Maximum loop count, adjustable up to 1024 | | `parallelism` | number | 0 | Parallelism, 0 = serial, maximum 20 (default concurrency 5) | ### foreach Field | Field | Type | Description | |------|------|------| | `foreach.nodes` | array | List of child nodes within the traverse body | | `foreach.flow` | array | Dependency relationships of child nodes within the traverse body | ## Full Example Assignment node outputs `"10,20,30"`; the for-each internal Shell node processes each item: ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "recurrence": "Normal", "name": "my_foreach", "script": { "path": "my_foreach", "runtime": { "command": "CONTROLLER_TRAVERSE" }, "content": "{\"maxIterations\":128,\"parallelism\":0}" }, "foreach": { "nodes": [ { "name": "inner_task", "script": { "path": "my_foreach/inner_task", "runtime": { "command": "DIDE_SHELL" }, "content": "echo \"processing item: ${dag.foreach.current}\"" } } ], "flow": [ { "nodeId": "inner_task", "depends": [] } ] }, "outputs": { "nodeOutputs": [ { "data": "${projectIdentifier}.my_foreach", "artifactType": "NodeOutput" } ] } } ], "flow": [ { "nodeId": "my_foreach", "depends": [ { "type": "Normal", "output": "${projectIdentifier}_root" } ] } ] } } ``` ## Restrictions | Restriction | Details | |--------|------| | Maximum iterations | Default 128, maximum 1024 | | Concurrency | Maximum 20 | | Test method | Cannot be run directly in Data Studio; must be published and smoke tested in the Operations Center | | Standalone execution | Standalone smoke testing, backfill, and manual execution are not supported | | Internal branches | If branch nodes are used, all branches must converge at a merge node before connecting to the end node | | Re-run behavior | Automatic re-run on failure resumes from the failed node; manual re-run triggers a complete re-run | FILE:references/nodetypes/controller/CONTROLLER_TRAVERSE_END.md # Traverse End Node（CONTROLLER_TRAVERSE_END） ## Overview - Compute engine: `GENERAL` - Content format: empty (no code) - Extension: `.for-each-end` - Description: for-each traverse end (auto-generated) ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_controller_traverse_end", "script": { "path": "example_controller_traverse_end", "runtime": { "command": "CONTROLLER_TRAVERSE_END" }, "content": "" } } ] } } ``` FILE:references/nodetypes/controller/CONTROLLER_TRAVERSE_START.md # Traverse Start Node（CONTROLLER_TRAVERSE_START） ## Overview - Compute engine: `GENERAL` - Content format: empty (no code) - Extension: `.for-each-start` - Description: for-each traverse start (auto-generated) ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_controller_traverse_start", "script": { "path": "example_controller_traverse_start", "runtime": { "command": "CONTROLLER_TRAVERSE_START" }, "content": "" } } ] } } ``` FILE:references/nodetypes/controller/PARAM_HUB.md # Parameter Node（PARAM_HUB） ## Overview - Compute engine: `GENERAL` - Content format: empty (no code) - Extension: `.param-hub.json` The parameter node is a virtual node that does not perform data computation. It is used to centrally manage parameters; downstream nodes obtain required parameters by depending on the parameter node. **Use cases**: - Cross-node parameter passing: When multiple downstream nodes need upstream output parameters, the parameter node serves as a unified relay - Parameter management: Centrally manage constant and variable parameters; downstream nodes can obtain them by simply adding a dependency ## Parameter Types | Type | Description | Value | |------|------|------| | Constant | Fixed value parameter | Custom fixed value | | Variable | Scheduling variable parameter | Scheduling parameter expression (e.g., time variable) | | Pass-through variable | Passes upstream node output parameters | Binds to upstream node output parameters | ## Parameter Passing Rules - Task nodes referencing the parameter node **must be direct downstream nodes of the parameter node** - Downstream nodes bind the parameter node's output parameters in their scheduling parameters - In the downstream node script, reference parameter values via `${parameter_name}` ## Configuration Method ### Defining Parameters in the Parameter Node Define output parameters via `outputs.variables`: ```json { "name": "my_param_hub", "script": { "path": "my_param_hub", "runtime": { "command": "PARAM_HUB" }, "content": "" }, "outputs": { "variables": [ { "artifactType": "Variable", "name": "bizdate", "scope": "NodeContext", "type": "Constant", "value": "20260101" }, { "artifactType": "Variable", "name": "env", "scope": "NodeContext", "type": "Constant", "value": "prod" } ], "nodeOutputs": [ { "data": "${projectIdentifier}.my_param_hub", "artifactType": "NodeOutput" } ] } } ``` ### Downstream Node Parameter Reference Downstream nodes bind to the parameter node's output via `inputs.variables`: ```json { "name": "downstream_node", "script": { "path": "downstream_node", "runtime": { "command": "ODPS_SQL" }, "content": "SELECT * FROM my_table WHERE dt='${bizdate}' AND env='${env}';", "parameters": [ { "artifactType": "Variable", "name": "bizdate", "scope": "NodeContext", "type": "NodeOutput", "value": "${bizdate}", "node": { "output": "${projectIdentifier}.my_param_hub" } }, { "artifactType": "Variable", "name": "env", "scope": "NodeContext", "type": "NodeOutput", "value": "${env}", "node": { "output": "${projectIdentifier}.my_param_hub" } } ] }, "inputs": { "variables": [ { "artifactType": "Variable", "name": "bizdate", "scope": "NodeContext", "type": "NodeOutput", "value": "${bizdate}", "node": { "output": "${projectIdentifier}.my_param_hub" } }, { "artifactType": "Variable", "name": "env", "scope": "NodeContext", "type": "NodeOutput", "value": "${env}", "node": { "output": "${projectIdentifier}.my_param_hub" } } ], "nodeOutputs": [ { "data": "${projectIdentifier}.my_param_hub", "artifactType": "NodeOutput" } ] } } ``` ## Restrictions | Restriction | Details | |--------|------| | Pass level | Can only pass to direct downstream nodes | | Node type | Virtual node; does not perform computation | ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_param_hub", "script": { "path": "example_param_hub", "runtime": { "command": "PARAM_HUB" }, "content": "" } } ] } } ``` FILE:references/nodetypes/controller/SCHEDULER_TRIGGER.md # Trigger Node（SCHEDULER_TRIGGER） ## Overview - Compute engine: `GENERAL` - Content format: empty (no code) - Extension: `.json` - Description: HTTP trigger node, no code ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_scheduler_trigger", "script": { "path": "example_scheduler_trigger", "runtime": { "command": "SCHEDULER_TRIGGER" }, "content": "" } } ] } } ``` FILE:references/nodetypes/custom/CUSTOM.md # Custom Node（CUSTOM） ## Overview - Code: `9999` - Compute engine: `CUSTOM` - Content format: json - Extension: `.json` - LabelType: `DATA_PROCESS` - Description: Custom node type for extending node types not natively supported by DataWorks Custom nodes allow users to define task types not natively supported by the DataWorks platform through JSON configuration. They are suitable for scenarios that require integration with third-party systems, custom processing logic, or special scheduling needs. The node content is a free-format JSON object; the specific structure is defined by the user based on business requirements. ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_custom", "script": { "path": "example_custom", "runtime": { "command": "CUSTOM" }, "content": "{}" } } ] } } ``` ## Usage Notes - `script.content` is a custom JSON string; the structure is not strictly constrained and depends on the specific business scenario. - `script.runtime.command` is fixed as `CUSTOM`. - Suitable for scenarios involving integration with external systems or extending platform capabilities; requires use with a custom resource group and runtime environment. FILE:references/nodetypes/data_integration/DATAX.md # DataX Data Sync（DATAX） ## Overview - Code: `4` - Compute engine: `DI` - Content format: json - Extension: `.json` - LabelType: `DATA_PROCESS` - Description: DataX Data sync task (legacy) DataX is an earlier version of the offline data sync node type. Its content structure is similar to DI nodes, using the DIJob JSON format. For new projects, it is recommended to use the DI (code 23) node type instead. ## Content Structure `script.content` is a DIJob JSON string containing the following required fields: | Field | Type | Required | Description | |------|------|------|------| | `type` | string | Yes | Fixed value `"job"` | | `version` | string | Yes | Version number, recommended `"2.0"` | | `steps` | array | Yes | Steps array, contains Reader and Writer | | `order` | object | Yes | Step execution order | ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_datax", "script": { "path": "example_datax", "runtime": { "command": "DATAX" }, "content": "{\"type\":\"job\",\"version\":\"2.0\",\"steps\":[],\"order\":{\"hops\":[]},\"setting\":{\"speed\":{\"concurrent\":1}}}" } } ] } } ``` ## Restrictions - DATAX is a legacy node type. For new projects, it is recommended to use DI nodes instead. - The content structure and configuration are largely consistent with DI nodes. For Reader/Writer configuration, refer to the [DI Data Sync Development Guide](../di-guide.md). FILE:references/nodetypes/data_integration/DATAX2.md # DataX2 Data Sync（DATAX2） ## Overview - Code: `20` - Compute engine: `DI` - Content format: json - Extension: `.json` - LabelType: `DATA_PROCESS` - Description: DataX2 Data sync task DataX2 is the upgraded version of DataX. Its content structure is similar to DI nodes, using the DIJob JSON format. For new projects, it is recommended to use the DI (code 23) node type instead. ## Content Structure `script.content` is a DIJob JSON string containing the following required fields: | Field | Type | Required | Description | |------|------|------|------| | `type` | string | Yes | Fixed value `"job"` | | `version` | string | Yes | Version number, recommended `"2.0"` | | `steps` | array | Yes | Steps array, contains Reader and Writer | | `order` | object | Yes | Step execution order | ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_datax2", "script": { "path": "example_datax2", "runtime": { "command": "DATAX2" }, "content": "{\"type\":\"job\",\"version\":\"2.0\",\"steps\":[],\"order\":{\"hops\":[]},\"setting\":{\"speed\":{\"concurrent\":1}}}" } } ] } } ``` ## Restrictions - DATAX2 is a transitional node type. For new projects, it is recommended to use DI nodes instead. - The content structure and configuration are largely consistent with DI nodes. For Reader/Writer configuration, refer to the [DI Data Sync Development Guide](../di-guide.md). FILE:references/nodetypes/data_integration/DD_MERGE.md # Data Merge（DD_MERGE） ## Overview - Code: `222` - Compute engine: `DI` - Content format: json - Extension: `.json` - LabelType: `DATA_PROCESS` - Description: Data merge task DD_MERGE is used to merge data from multiple data sources or datasets into a unified dataset. The content is a JSON configuration object that defines the sources, targets, and merge rules. ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_dd_merge", "script": { "path": "example_dd_merge", "runtime": { "command": "DD_MERGE" }, "content": "{}" } } ] } } ``` FILE:references/nodetypes/data_integration/DI.md # Data Integration - Offline Sync（DI） ## Overview - Code: `23` - Compute engine: `DI` - Content format: json - Extension: `.json` - LabelType: `DATA_PROCESS` - Description: Offline data sync task (DIJob), the recommended data integration node type DI (Data Integration) is the offline data sync service of DataWorks, providing stable, efficient, and elastically scalable data transfer capabilities between heterogeneous data sources. It supports both wizard mode (visual configuration) and script mode (JSON script) for development. Help documentation: https://help.aliyun.com/zh/dataworks/user-guide/develop-an-offline-synchronization-node ## Features - **Sync mode**: Supports full sync, incremental sync, and combined full-incremental sync - **Sync range**: Supports single-table sync, whole-database sync, and sharded-database/sharded-table sync - **Data sources**: Supports dozens of data sources including MySQL, MaxCompute (ODPS), Hologres, PostgreSQL, Oracle, SQL Server, OSS, FTP, Kafka, Elasticsearch, etc. - **Field mapping**: Supports column mapping configuration between source and target - **Concurrency control**: Supports configuring the number of parallel read/write channels - **Speed limit**: Supports transfer speed throttling - **Dirty data handling**: Supports setting a dirty data record count threshold - **Transfer semantics**: At-least-once delivery (may produce duplicates); exactly-once is not supported ## Content Structure（DIJob JSON） `script.content` is a DIJob JSON string with the following top-level structure: | Field | Type | Required | Description | |------|------|------|------| | `type` | string | Yes | Fixed value `"job"` | | `version` | string | Yes | Version number, recommended `"2.0"` | | `steps` | array | Yes | Steps array, must contain at least one Reader and one Writer | | `order` | object | Yes | Step execution order, defines from/to relationships via `hops` | | `setting` | object | No | Runtime parameters (concurrency count, speed limit, dirty data threshold, etc.) | Each step contains `stepType` (data source type), `name` (step name), `category` (`reader` or `writer`), and `parameter` (data source parameters). ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_di", "script": { "path": "example_di", "runtime": { "command": "DI" }, "content": "{\"type\":\"job\",\"version\":\"2.0\",\"steps\":[],\"order\":{\"hops\":[]},\"setting\":{\"speed\":{\"concurrent\":1}}}" } } ] } } ``` ## Restrictions - The number of columns in the Reader and Writer must be consistent, mapped one-to-one in order. - The data source name in `parameter.datasource` must exactly match the data source name registered in DataWorks. - The DI node spec does not require a `datasource` field; data source information is configured in `parameter.datasource` within the code JSON. - Setting `concurrent` too high may put pressure on the source database; adjust according to actual conditions. - When dirty data causes a task failure, data that has already been successfully written will not be rolled back. - For production environments, it is recommended to set `errorLimit.record` to 0 to ensure data quality. For detailed Reader/Writer configuration, refer to the [DI Data Sync Development Guide](../di-guide.md). FILE:references/nodetypes/data_integration/DT.md # DT Data Sync（DT） ## Overview - Code: `21` - Compute engine: `DI` - Content format: json - Extension: `.json` - LabelType: `DATA_PROCESS` - Description: DT Data sync task DT is a data sync node type in DataWorks Data Integration. The content is a JSON configuration object used to define data transfer tasks. ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_dt", "script": { "path": "example_dt", "runtime": { "command": "DT" }, "content": "{}" } } ] } } ``` FILE:references/nodetypes/data_integration/RI.md # Real-time Data Sync（RI） ## Overview - Code: `900` - Compute engine: `DI` - Content format: json - Extension: `.json` - LabelType: `DATA_PROCESS` - Description: Real-time data sync task RI (Real-time Integration) is the real-time data sync service of DataWorks, supporting second-level latency CDC (Change Data Capture) to capture source data changes and sync them to the target in real time. Unlike offline sync (DI) which uses batch periodic scheduling, RI runs continuously in streaming mode, suitable for scenarios requiring high data timeliness. Help documentation: https://help.aliyun.com/zh/dataworks/user-guide/create-and-manage-real-time-synchronization-nodes ## Features - **Real-time sync**: Second-level latency, continuously captures source data changes - **Sync range**: Supports single-table real-time sync and whole-database real-time sync - **Data sources**: Supports real-time sync from relational databases such as MySQL, PostgreSQL, Oracle, SQL Server, etc. to targets such as MaxCompute, Hologres, Kafka, Elasticsearch, etc. ## Content Structure `script.content` is a DIJob JSON string with the same structure as DI nodes: | Field | Type | Required | Description | |------|------|------|------| | `type` | string | Yes | Fixed value `"job"` | | `version` | string | Yes | Version number, recommended `"2.0"` | | `steps` | array | Yes | Steps array, contains Reader and Writer | | `order` | object | Yes | Step execution order | | `setting` | object | No | Runtime parameters configuration | ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_ri", "script": { "path": "example_ri", "runtime": { "command": "RI" }, "content": "{\"type\":\"job\",\"version\":\"2.0\",\"steps\":[],\"order\":{\"hops\":[]},\"setting\":{\"speed\":{\"concurrent\":1}}}" } } ] } } ``` ## Restrictions - Real-time sync tasks are long-running tasks, which differ from the periodic scheduling approach of offline sync. - Must be used with CDC-capable data sources (e.g., MySQL binlog, PostgreSQL WAL, etc.). - The Reader/Writer configuration is similar to DI nodes. For details, refer to the [DI Data Sync Development Guide](../di-guide.md). FILE:references/nodetypes/data_integration/TT_MERGE.md # Table Merge（TT_MERGE） ## Overview - Code: `200` - Compute engine: `DI` - Content format: json - Extension: `.json` - LabelType: `DATA_PROCESS` - Description: Table merge task TT_MERGE is used to merge data from multiple tables and write it into a single target table. The content is a JSON configuration object that defines the source tables, target table, and merge strategy. ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_tt_merge", "script": { "path": "example_tt_merge", "runtime": { "command": "TT_MERGE" }, "content": "{}" } } ] } } ``` FILE:references/nodetypes/database/ADB_for_MySQL.md # AnalyticDB for MySQL（ADB_for_MySQL） ## Overview - Compute engine: `DATABASE` - Content format: sql - Extension: `.sql` - Data source type: `adb_mysql` - Description: Execute SQL statements against AnalyticDB for MySQL database Used to directly execute SQL queries and data processing against AnalyticDB for MySQL（ADB MySQL）cloud-native data warehouse to execute SQL queries and data processing. The corresponding data source type must be registered in the workspace beforehand. ## Prerequisites - The data source type has been added to the DataWorks workspace AnalyticDB for MySQL type data source - The data source connectivity test has passed ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_adb_for_mysql", "script": { "path": "example_adb_for_mysql", "runtime": { "command": "ADB for MySQL" }, "content": "SELECT 1;" } } ] } } ``` ## Restrictions - SQL syntax must comply with AnalyticDB for MySQL database specifications（MySQL-syntax compatible) - Mixing DDL and DML statements in a single node is not supported (separate execution recommended) - Execution timeout is limited by the data source configuration FILE:references/nodetypes/database/ADB_for_PostgreSQL.md # AnalyticDB for PostgreSQL（ADB_for_PostgreSQL） ## Overview - Compute engine: `DATABASE` - Content format: sql - Extension: `.sql` - Data source type: `adb_pg` - Description: Execute SQL statements against AnalyticDB for PostgreSQL database Used to directly execute SQL queries and data processing against AnalyticDB for PostgreSQL（ADB PG）cloud-native data warehouse to execute SQL queries and data processing. The corresponding data source type must be registered in the workspace beforehand. ## Prerequisites - The data source type has been added to the DataWorks workspace AnalyticDB for PostgreSQL type data source - The data source connectivity test has passed ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_adb_for_postgresql", "script": { "path": "example_adb_for_postgresql", "runtime": { "command": "ADB for PostgreSQL" }, "content": "SELECT 1;" } } ] } } ``` ## Restrictions - SQL syntax must comply with AnalyticDB for PostgreSQL database specifications（PostgreSQL-syntax compatible, with Greenplum extension support) - Mixing DDL and DML statements in a single node is not supported (separate execution recommended) - Execution timeout is limited by the data source configuration FILE:references/nodetypes/database/CLICK_SQL.md # ClickHouse SQL（CLICK_SQL） ## Overview - Compute engine: `CLICKHOUSE` - Content format: sql - Extension: `.sql` - Data source type: `clickhouse` - Description: Execute SQL statements against ClickHouse database Used to directly execute SQL queries and data processing against ClickHouse columnar storage database to execute SQL queries and data processing. The corresponding data source type must be registered in the workspace beforehand. > Note: The compute engine for ClickHouse nodes is `CLICKHOUSE`, which differs from the `DATABASE` engine used by other database-type nodes. ## Prerequisites - The data source type has been added to the DataWorks workspace ClickHouse type data source - The data source connectivity test has passed - A ClickHouse cluster has been provisioned and network connectivity is confirmed ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_click_sql", "script": { "path": "example_click_sql", "runtime": { "command": "CLICK_SQL" }, "content": "SELECT 1;" } } ] } } ``` ## Supported SQL Operations - **DDL**: CREATE TABLE, DROP TABLE, ALTER TABLE, CREATE VIEW, etc. - **DML**: INSERT INTO, SELECT ... INTO, etc. - **Queries**: SELECT (supports JOIN, subqueries, aggregate functions, window functions, etc.) - **ClickHouse-specific syntax**: MergeTree family engine table creation, MATERIALIZED VIEW, partition operations, etc. ## Restrictions - SQL syntax must comply with ClickHouse SQL specifications - ClickHouse does not support standard UPDATE and DELETE statements; use ALTER TABLE ... UPDATE/DELETE (Mutation operations) instead - Only one SQL statement can be executed at a time - Execution timeout is limited by the data source configuration - ClickHouse has limited transaction support; pay attention to data consistency ## Reference - [ClickHouse SQL Node Documentation](https://help.aliyun.com/zh/dataworks/user-guide/clickhouse-sql-node) FILE:references/nodetypes/database/DB2.md # IBM DB2（DB2） ## Overview - Compute engine: `DATABASE` - Content format: sql - Extension: `.sql` - Data source type: `db2` - Description: Execute SQL statements against IBM DB2 database Used to directly execute SQL queries and data processing against IBM DB2 database to execute SQL queries and data processing. The corresponding data source type must be registered in the workspace beforehand. ## Prerequisites - The data source type has been added to the DataWorks workspace DB2 type data source - The data source connectivity test has passed ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_db2", "script": { "path": "example_db2", "runtime": { "command": "DB2" }, "content": "SELECT 1 FROM SYSIBM.SYSDUMMY1;" } } ] } } ``` ## Restrictions - SQL syntax must comply with IBM DB2 database specifications - DB2 uses the `SYSIBM.SYSDUMMY1` table instead of the `DUAL` table used in other databases (e.g., `SELECT 1 FROM SYSIBM.SYSDUMMY1`) - Mixing DDL and DML statements in a single node is not supported (separate execution recommended) - Execution timeout is limited by the data source configuration FILE:references/nodetypes/database/DRDS.md # PolarDB-X / DRDS（DRDS） ## Overview - Compute engine: `DATABASE` - Content format: sql - Extension: `.sql` - Data source type: `drds` - Description: Execute SQL statements against PolarDB-X (DRDS) database Used to directly execute SQL queries and data processing against PolarDB-X (formerly DRDS) distributed database. The corresponding data source type must be registered in the workspace beforehand. ## Prerequisites - The data source type has been added to the DataWorks workspace DRDS type data source - The data source connectivity test has passed ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_drds", "script": { "path": "example_drds", "runtime": { "command": "DRDS" }, "content": "SELECT 1;" } } ] } } ``` ## Restrictions - SQL syntax must comply with PolarDB-X（DRDS）database specifications - Mixing DDL and DML statements in a single node is not supported (separate execution recommended) - Execution timeout is limited by the data source configuration FILE:references/nodetypes/database/Doris.md # Doris（Doris） ## Overview - Compute engine: `DATABASE` - Content format: sql - Extension: `.sql` - Data source type: `doris` - Description: Execute SQL statements against Doris database Used to directly execute SQL queries and data processing against Apache Doris real-time analytical database to execute SQL queries and data processing. The corresponding data source type must be registered in the workspace beforehand. ## Prerequisites - The data source type has been added to the DataWorks workspace Doris type data source - The data source connectivity test has passed ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_doris", "script": { "path": "example_doris", "runtime": { "command": "Doris" }, "content": "SELECT 1;" } } ] } } ``` ## Restrictions - SQL syntax must comply with Apache Doris database specifications（MySQL-protocol compatible) - Mixing DDL and DML statements in a single node is not supported (separate execution recommended) - Execution timeout is limited by the data source configuration FILE:references/nodetypes/database/MYSQL.md # MySQL（MYSQL） ## Overview - Compute engine: `DATABASE` - Content format: sql - Extension: `.sql` - Data source type: `mysql` - Description: Execute SQL statements against MySQL database Used to directly execute SQL queries and data processing against MySQL database to execute SQL queries and data processing. The corresponding data source type must be registered in the workspace beforehand. ## Prerequisites - The data source type has been added to the DataWorks workspace MySQL type data source - The data source connectivity test has passed ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_mysql", "script": { "path": "example_mysql", "runtime": { "command": "MySQL" }, "content": "SELECT 1;" } } ] } } ``` ## Restrictions - SQL syntax must comply with MySQL database specifications - Mixing DDL and DML statements in a single node is not supported (separate execution recommended) - Execution timeout is limited by the data source configuration FILE:references/nodetypes/database/Mariadb.md # MariaDB（Mariadb） ## Overview - Compute engine: `DATABASE` - Content format: sql - Extension: `.sql` - Data source type: `mariadb` - Description: Execute SQL statements against MariaDB database Used to directly execute SQL queries and data processing against MariaDB database to execute SQL queries and data processing. The corresponding data source type must be registered in the workspace beforehand. ## Prerequisites - The data source type has been added to the DataWorks workspace MariaDB type data source - The data source connectivity test has passed ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_mariadb", "script": { "path": "example_mariadb", "runtime": { "command": "Mariadb" }, "content": "SELECT 1;" } } ] } } ``` ## Restrictions - SQL syntax must comply with MariaDB database specifications（highly MySQL-compatible) - Mixing DDL and DML statements in a single node is not supported (separate execution recommended) - Execution timeout is limited by the data source configuration FILE:references/nodetypes/database/OceanBase.md # OceanBase（OceanBase） ## Overview - Compute engine: `DATABASE` - Content format: sql - Extension: `.sql` - Data source type: `oceanbase` - Description: Execute SQL statements against OceanBase database Used to directly execute SQL queries and data processing against OceanBase distributed database to execute SQL queries and data processing. The corresponding data source type must be registered in the workspace beforehand. ## Prerequisites - The data source type has been added to the DataWorks workspace OceanBase type data source - The data source connectivity test has passed ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_oceanbase", "script": { "path": "example_oceanbase", "runtime": { "command": "OceanBase" }, "content": "SELECT 1;" } } ] } } ``` ## Restrictions - SQL syntax must comply with OceanBase database specifications (compatible with MySQL or Oracle mode) - Mixing DDL and DML statements in a single node is not supported (separate execution recommended) - Execution timeout is limited by the data source configuration FILE:references/nodetypes/database/Oracle.md # Oracle（Oracle） ## Overview - Compute engine: `DATABASE` - Content format: sql - Extension: `.sql` - Data source type: `oracle` - Description: Execute SQL statements against Oracle database Used to directly execute SQL queries and data processing against Oracle database to execute SQL queries and data processing. The corresponding data source type must be registered in the workspace beforehand. ## Prerequisites - The data source type has been added to the DataWorks workspace Oracle type data source - The data source connectivity test has passed ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_oracle", "script": { "path": "example_oracle", "runtime": { "command": "Oracle" }, "content": "SELECT 1;" } } ] } } ``` ## Restrictions - SQL syntax must comply with Oracle database specifications（PL/SQL） - Mixing DDL and DML statements in a single node is not supported (separate execution recommended) - Execution timeout is limited by the data source configuration FILE:references/nodetypes/database/POSTGRESQL.md # PostgreSQL（POSTGRESQL） ## Overview - Compute engine: `DATABASE` - Content format: sql - Extension: `.sql` - Data source type: `postgresql` - Description: Execute SQL statements against PostgreSQL database Used to directly execute SQL queries and data processing against PostgreSQL database to execute SQL queries and data processing. The corresponding data source type must be registered in the workspace beforehand. ## Prerequisites - The data source type has been added to the DataWorks workspace PostgreSQL type data source - The data source connectivity test has passed ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_postgresql", "script": { "path": "example_postgresql", "runtime": { "command": "POSTGRESQL" }, "content": "SELECT 1;" } } ] } } ``` ## Restrictions - SQL syntax must comply with PostgreSQL database specifications - Mixing DDL and DML statements in a single node is not supported (separate execution recommended) - Execution timeout is limited by the data source configuration FILE:references/nodetypes/database/Redshift.md # Amazon Redshift（Redshift） ## Overview - Compute engine: `DATABASE` - Content format: sql - Extension: `.sql` - Data source type: `redshift` - Description: Execute SQL statements against Amazon Redshift data warehouse Used to directly execute SQL queries and data processing against Amazon Redshift cloud data warehouse. The corresponding data source type must be registered in the workspace beforehand. ## Prerequisites - The data source type has been added to the DataWorks workspace Redshift type data source - The data source connectivity test has passed - Network connectivity is confirmed (the Redshift cluster must be accessible from DataWorks) ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_redshift", "script": { "path": "example_redshift", "runtime": { "command": "Redshift" }, "content": "SELECT 1;" } } ] } } ``` ## Restrictions - SQL syntax must comply with Amazon Redshift database specifications(based on PostgreSQL) - Mixing DDL and DML statements in a single node is not supported (separate execution recommended) - Execution timeout is limited by the data source configuration FILE:references/nodetypes/database/SQLSERVER.md # SQL Server（SQLSERVER） ## Overview - Compute engine: `DATABASE` - Content format: sql - Extension: `.sql` - Data source type: `sqlserver` - Description: Execute SQL statements against SQL Server database Used to directly execute SQL queries and data processing against SQL Server database to execute SQL queries and data processing. The corresponding data source type must be registered in the workspace beforehand. ## Prerequisites - The data source type has been added to the DataWorks workspace SQL Server type data source - The data source connectivity test has passed ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_sqlserver", "script": { "path": "example_sqlserver", "runtime": { "command": "Sql Server" }, "content": "SELECT 1;" } } ] } } ``` ## Restrictions - SQL syntax must comply with SQL Server（T-SQL）database specifications - Mixing DDL and DML statements in a single node is not supported (separate execution recommended) - Execution timeout is limited by the data source configuration FILE:references/nodetypes/database/Saphana.md # SAP HANA（Saphana） ## Overview - Compute engine: `DATABASE` - Content format: sql - Extension: `.sql` - Data source type: `saphana` - Description: Execute SQL statements against SAP HANA database Used to directly execute SQL queries and data processing against SAP HANA in-memory database to execute SQL queries and data processing. The corresponding data source type must be registered in the workspace beforehand. ## Prerequisites - The data source type has been added to the DataWorks workspace SAP HANA type data source - The data source connectivity test has passed ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_saphana", "script": { "path": "example_saphana", "runtime": { "command": "Saphana" }, "content": "SELECT 1 FROM DUMMY;" } } ] } } ``` ## Restrictions - SQL syntax must comply with SAP HANA SQL specifications - SAP HANA uses the `DUMMY` table instead of the `DUAL` table used in other databases (e.g., `SELECT 1 FROM DUMMY`) - Mixing DDL and DML statements in a single node is not supported (separate execution recommended) - Execution timeout is limited by the data source configuration FILE:references/nodetypes/database/Selectdb.md # SelectDB（Selectdb） ## Overview - Compute engine: `DATABASE` - Content format: sql - Extension: `.sql` - Data source type: `selectdb` - Description: Execute SQL statements against SelectDB database Used to directly execute SQL queries and data processing against SelectDB cloud-native real-time data warehouse to execute SQL queries and data processing. The corresponding data source type must be registered in the workspace beforehand. ## Prerequisites - The data source type has been added to the DataWorks workspace SelectDB type data source - The data source connectivity test has passed ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_selectdb", "script": { "path": "example_selectdb", "runtime": { "command": "Selectdb" }, "content": "SELECT 1;" } } ] } } ``` ## Restrictions - SQL syntax must comply with SelectDB database specifications(based on Apache Doris, MySQL-protocol compatible) - Mixing DDL and DML statements in a single node is not supported (separate execution recommended) - Execution timeout is limited by the data source configuration FILE:references/nodetypes/database/StarRocks.md # StarRocks（StarRocks） ## Overview - Compute engine: `DATABASE` - Content format: sql - Extension: `.sql` - Data source type: `starrocks` - Description: Execute SQL statements against StarRocks database Used to directly execute SQL queries and data processing against StarRocks real-time analytical database to execute SQL queries and data processing. The corresponding data source type must be registered in the workspace beforehand. ## Prerequisites - The data source type has been added to the DataWorks workspace StarRocks type data source - The data source connectivity test has passed ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_starrocks", "script": { "path": "example_starrocks", "runtime": { "command": "StarRocks" }, "content": "SELECT 1;" } } ] } } ``` ## Restrictions - SQL syntax must comply with StarRocks database specifications（MySQL-protocol compatible) - Mixing DDL and DML statements in a single node is not supported (separate execution recommended) - Execution timeout is limited by the data source configuration FILE:references/nodetypes/database/Vertica.md # Vertica（Vertica） ## Overview - Compute engine: `DATABASE` - Content format: sql - Extension: `.sql` - Data source type: `vertica` - Description: Execute SQL statements against Vertica database Used to directly execute SQL queries and data processing against Vertica columnar analytical database to execute SQL queries and data processing. The corresponding data source type must be registered in the workspace beforehand. ## Prerequisites - The data source type has been added to the DataWorks workspace Vertica type data source - The data source connectivity test has passed ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_vertica", "script": { "path": "example_vertica", "runtime": { "command": "Vertica" }, "content": "SELECT 1;" } } ] } } ``` ## Restrictions - SQL syntax must comply with Vertica database specifications - Mixing DDL and DML statements in a single node is not supported (separate execution recommended) - Execution timeout is limited by the data source configuration FILE:references/nodetypes/emr/EMR_FILE.md # EMR File Resource（EMR_FILE） ## Overview - Compute engine: `EMR` - Content format: json - Extension: `.json` - Data source type: `emr` - Code: 232 - LabelType：RESOURCE - Description: Manage file resources used by EMR clusters in DataWorks Used to register and manage general file resources (such as configuration files, data files, etc.) used on EMR clusters. These can be referenced by EMR nodes and are suitable for scenarios that require external files in EMR jobs. ## Prerequisites - An EMR cluster has been created and added as a DataWorks compute resource - The DataWorks resource group has network connectivity with the EMR cluster ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_emr_file", "script": { "path": "example_emr_file", "runtime": { "command": "EMR_FILE" }, "content": "{}" } } ] } } ``` FILE:references/nodetypes/emr/EMR_FUNCTION.md # EMR Function（EMR_FUNCTION） ## Overview - Compute engine: `EMR` - Content format: empty (no code) - Extension: none - Data source type: `emr` - Code: 262 - LabelType：FUNCTION - Description: Manage custom functions for EMR clusters in DataWorks Used to register and manage in DataWorks EMR custom functions (UDF/UDAF/UDTF) on the cluster. Once registered, they can be called directly in Hive SQL, Spark SQL, and other nodes. ## Prerequisites - An EMR cluster has been created and added as a DataWorks compute resource - The DataWorks resource group has network connectivity with the EMR cluster - The JAR resource (EMR_JAR) required by the function has been uploaded ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_emr_function", "script": { "path": "example_emr_function", "runtime": { "command": "EMR_FUNCTION" }, "content": "" } } ] } } ``` FILE:references/nodetypes/emr/EMR_HIVE.md # EMR Hive SQL（EMR_HIVE） ## Overview - Compute engine: `EMR` - Content format: sql - Extension: `.sql` - Data source type: `emr` - Code: 227 - Description: Run Hive SQL queries on EMR clusters Execute Hive SQL statements on Alibaba Cloud E-MapReduce clusters through DataWorks scheduling, suitable for Hive-based data warehouse ETL processing, data querying, and analysis scenarios. ## Prerequisites - An EMR cluster has been created and added as a DataWorks compute resource - The DataWorks resource group has network connectivity with the EMR cluster - EMR cluster has Hive service installed ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_emr_hive", "script": { "path": "example_emr_hive", "runtime": { "command": "EMR_HIVE" }, "content": "SELECT 1;" } } ] } } ``` ## Reference - [EMR Hive Node](https://help.aliyun.com/zh/dataworks/user-guide/emr-hive-node) FILE:references/nodetypes/emr/EMR_HIVE_CLI.md # EMR Hive CLI（EMR_HIVE_CLI） ## Overview - Compute engine: `EMR` - Content format: empty (no code) - Extension: none - Data source type: `emr` - Code: 265 - Description: Execute Hive operations on EMR clusters via Hive CLI command line Execute Hive commands on EMR clusters through the Hive command-line interface (CLI), suitable for scenarios that require Hive CLI-specific features or interactive commands. ## Prerequisites - An EMR cluster has been created and added as a DataWorks compute resource - The DataWorks resource group has network connectivity with the EMR cluster - EMR cluster has Hive service installed ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_emr_hive_cli", "script": { "path": "example_emr_hive_cli", "runtime": { "command": "EMR_HIVE_CLI" }, "content": "" } } ] } } ``` FILE:references/nodetypes/emr/EMR_IMPALA.md # EMR Impala SQL（EMR_IMPALA） ## Overview - Compute engine: `EMR` - Content format: sql - Extension: `.sql` - Data source type: `emr` - Code: 260 - Description: Run Impala SQL queries on EMR clusters Execute SQL statements on the Impala engine of EMR clusters through DataWorks scheduling, suitable for scenarios requiring low-latency interactive analysis. Impala is a massively parallel processing SQL engine based on in-memory computing, providing sub-second query responses. ## Prerequisites - An EMR cluster has been created and added as a DataWorks compute resource - The DataWorks resource group has network connectivity with the EMR cluster - EMR cluster has Impala service installed ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_emr_impala", "script": { "path": "example_emr_impala", "runtime": { "command": "EMR_IMPALA" }, "content": "SELECT 1;" } } ] } } ``` ## Reference - [EMR Impala Node](https://help.aliyun.com/zh/dataworks/user-guide/emr-impala-node) FILE:references/nodetypes/emr/EMR_JAR.md # EMR JAR Resource（EMR_JAR） ## Overview - Compute engine: `EMR` - Content format: json - Extension: `.json` - Data source type: `emr` - Code: 231 - LabelType：RESOURCE - Description: Manage JAR resources used by EMR clusters in DataWorks Used to register and manage JAR package resources used on EMR clusters. These can be referenced by EMR Spark, EMR MR, and other nodes, suitable for scenarios requiring additional JAR dependencies such as custom UDFs, Spark applications, etc. ## Prerequisites - An EMR cluster has been created and added as a DataWorks compute resource - The DataWorks resource group has network connectivity with the EMR cluster ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_emr_jar", "script": { "path": "example_emr_jar", "runtime": { "command": "EMR_JAR" }, "content": "{}" } } ] } } ``` FILE:references/nodetypes/emr/EMR_KYUUBI.md # EMR Kyuubi SQL（EMR_KYUUBI） ## Overview - Compute engine: `EMR` - Content format: sql - Extension: `.sql` - Data source type: `emr` - Code: 268 - Description: Run SQL queries via Kyuubi on EMR clusters Execute SQL statements on the Kyuubi service of EMR clusters through DataWorks scheduling. Kyuubi is a distributed multi-tenant Thrift/JDBC/ODBC service gateway that supports multiple backend engines such as Spark, Flink, Trino, etc., suitable for scenarios requiring a unified SQL entry point to access multiple compute engines. ## Prerequisites - An EMR cluster has been created and added as a DataWorks compute resource - The DataWorks resource group has network connectivity with the EMR cluster - EMR cluster has Kyuubi service installed ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_emr_kyuubi", "script": { "path": "example_emr_kyuubi", "runtime": { "command": "EMR_KYUUBI" }, "content": "SELECT 1;" } } ] } } ``` ## Reference - [EMR Kyuubi Node](https://help.aliyun.com/zh/dataworks/user-guide/emr-kyuubi-node) FILE:references/nodetypes/emr/EMR_MR.md # EMR MapReduce（EMR_MR） ## Overview - Compute engine: `EMR` - Content format: shell - Extension: `.sh` - Data source type: `emr` - Code: 230 - Description: Submit MapReduce jobs on EMR clusters Submit and run Hadoop MapReduce jobs on EMR clusters via shell scripts, suitable for big data batch processing scenarios using the traditional MapReduce programming model. ## Prerequisites - An EMR cluster has been created and added as a DataWorks compute resource - The DataWorks resource group has network connectivity with the EMR cluster - EMR cluster has Hadoop/YARN service installed ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_emr_mr", "script": { "path": "example_emr_mr", "runtime": { "command": "EMR_MR" }, "content": "#!/bin/bash\necho 'hello'" } } ] } } ``` ## Reference - [EMR MR Node](https://help.aliyun.com/zh/dataworks/user-guide/emr-mr-node) FILE:references/nodetypes/emr/EMR_PRESTO.md # EMR Presto SQL（EMR_PRESTO） ## Overview - Compute engine: `EMR` - Content format: sql - Extension: `.sql` - Data source type: `emr` - Code: 259 - Description: Run Presto SQL queries on EMR clusters Execute SQL statements on the Presto engine of EMR clusters through DataWorks scheduling, suitable for scenarios requiring interactive analysis and cross-data-source federated queries. Presto is a distributed SQL query engine that excels at low-latency ad-hoc queries. ## Prerequisites - An EMR cluster has been created and added as a DataWorks compute resource - The DataWorks resource group has network connectivity with the EMR cluster - EMR cluster has Presto service installed ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_emr_presto", "script": { "path": "example_emr_presto", "runtime": { "command": "EMR_PRESTO" }, "content": "SELECT 1;" } } ] } } ``` ## Reference - [EMR Presto Node](https://help.aliyun.com/zh/dataworks/user-guide/emr-presto-node) FILE:references/nodetypes/emr/EMR_PYSPARK.md # EMR PySpark（EMR_PYSPARK） ## Overview - Compute engine: `EMR` - Content format: python - Extension: `.py` - Data source type: `emr` - Code: 269 - Description: Run PySpark jobs on EMR clusters Write Python business logic in DataWorks and submit it to EMR clusters via spark-submit for execution. Suitable for distributed data processing scenarios using Python, such as machine learning, big data analysis, etc. Supports both EMR semi-managed and fully-managed (Serverless Spark) clusters. ## Prerequisites - An EMR cluster has been created and added as a DataWorks compute resource - The DataWorks resource group has network connectivity with the EMR cluster - EMR cluster has Spark service installed - Only Serverless resource groups are supported ## Restrictions - Only supports submitting the entire Python file as a single Spark job; running selected code snippets is not supported - Supports DataLake and Custom cluster types for EMR compute resources, as well as EMR Serverless Spark compute resources - Usage with semi-managed clusters requires submitting a ticket for evaluation ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_emr_pyspark", "script": { "path": "example_emr_pyspark", "runtime": { "command": "EMR_PYSPARK" }, "content": "print('hello')" } } ] } } ``` ## Reference - [EMR PySpark Node](https://help.aliyun.com/zh/dataworks/user-guide/emr-pyspark-node) FILE:references/nodetypes/emr/EMR_SCOOP.md # EMR Sqoop（EMR_SCOOP） ## Overview - Compute engine: `EMR` - Content format: empty (no code) - Extension: none - Data source type: `emr` - Code: 263 - Description: Run Sqoop data migration tasks on EMR clusters Execute data import and export tasks between relational databases and Hadoop on EMR clusters using the Sqoop tool, suitable for bulk data transfer scenarios between RDBMS and HDFS/Hive. ## Prerequisites - An EMR cluster has been created and added as a DataWorks compute resource - The DataWorks resource group has network connectivity with the EMR cluster - EMR cluster has Sqoop service installed - The target relational database has network connectivity with the EMR cluster ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_emr_scoop", "script": { "path": "example_emr_scoop", "runtime": { "command": "EMR_SCOOP" }, "content": "" } } ] } } ``` FILE:references/nodetypes/emr/EMR_SHELL.md # EMR Shell（EMR_SHELL） ## Overview - Compute engine: `EMR` - Content format: shell - Extension: `.sh` - Data source type: `emr` - Code: 257 - Description: Run shell scripts on EMR clusters Execute shell scripts on the Master node of EMR clusters, suitable for running system commands, file operations, cluster management, and other general shell tasks in the EMR cluster environment. ## Prerequisites - An EMR cluster has been created and added as a DataWorks compute resource - The DataWorks resource group has network connectivity with the EMR cluster ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_emr_shell", "script": { "path": "example_emr_shell", "runtime": { "command": "EMR_SHELL" }, "content": "#!/bin/bash\necho 'hello'" } } ] } } ``` ## Reference - [EMR Shell Node](https://help.aliyun.com/zh/dataworks/user-guide/emr-shell-node) FILE:references/nodetypes/emr/EMR_SPARK.md # EMR Spark Job（EMR_SPARK） ## Overview - Compute engine: `EMR` - Content format: shell - Extension: `.sh` - Data source type: `emr` - Code: 228 - Description: Submit Spark jobs on EMR clusters Write spark-submit commands via shell scripts to submit and run Spark jobs (such as Spark applications in JAR package form) on EMR clusters, suitable for submitting and scheduling custom Spark applications. ## Prerequisites - An EMR cluster has been created and added as a DataWorks compute resource - The DataWorks resource group has network connectivity with the EMR cluster - EMR cluster has Spark service installed ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_emr_spark", "script": { "path": "example_emr_spark", "runtime": { "command": "EMR_SPARK" }, "content": "#!/bin/bash\necho 'hello'" } } ] } } ``` ## Reference - [EMR Spark Node](https://help.aliyun.com/zh/dataworks/user-guide/emr-spark-node) FILE:references/nodetypes/emr/EMR_SPARK_SHELL.md # EMR Spark Shell（EMR_SPARK_SHELL） ## Overview - Compute engine: `EMR` - Content format: shell - Extension: `.sh` - Data source type: `emr` - Code: 258 - Description: Run Spark Shell scripts on EMR clusters Execute Spark Shell interactive commands on EMR clusters via shell scripts, suitable for scenarios that require using Spark Shell (Scala) for data exploration and processing. ## Prerequisites - An EMR cluster has been created and added as a DataWorks compute resource - The DataWorks resource group has network connectivity with the EMR cluster - EMR cluster has Spark service installed ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_emr_spark_shell", "script": { "path": "example_emr_spark_shell", "runtime": { "command": "EMR_SPARK_SHELL" }, "content": "#!/bin/bash\necho 'hello'" } } ] } } ``` FILE:references/nodetypes/emr/EMR_SPARK_SQL.md # EMR Spark SQL（EMR_SPARK_SQL） ## Overview - Compute engine: `EMR` - Content format: sql - Extension: `.sql` - Data source type: `emr` - Code: 229 - Description: Run Spark SQL queries on EMR clusters Execute SQL statements on the Spark SQL engine of EMR clusters through DataWorks scheduling, suitable for data processing and analysis scenarios using Spark SQL. Spark SQL is compatible with Hive SQL syntax while providing richer built-in functions and better performance. ## Prerequisites - An EMR cluster has been created and added as a DataWorks compute resource - The DataWorks resource group has network connectivity with the EMR cluster - EMR cluster has Spark service installed ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_emr_spark_sql", "script": { "path": "example_emr_spark_sql", "runtime": { "command": "EMR_SPARK_SQL" }, "content": "SELECT 1;" } } ] } } ``` ## Reference - [EMR Spark SQL Node](https://help.aliyun.com/zh/dataworks/user-guide/emr-spark-sql-node) FILE:references/nodetypes/emr/EMR_SPARK_STREAMING.md # EMR Spark Streaming（EMR_SPARK_STREAMING） ## Overview - Compute engine: `EMR` - Content format: shell - Extension: `.sh` - Data source type: `emr` - Code: 264 - Description: Submit Spark Streaming stream processing jobs on EMR clusters Submit Spark Streaming jobs on EMR clusters via shell scripts, suitable for real-time or near-real-time stream data processing scenarios such as log collection and analysis, real-time metric computation, etc. ## Prerequisites - An EMR cluster has been created and added as a DataWorks compute resource - The DataWorks resource group has network connectivity with the EMR cluster - EMR cluster has Spark service installed ## Restrictions - Spark Streaming jobs are long-running tasks; ensure that the scheduling period and timeout are configured appropriately ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_emr_spark_streaming", "script": { "path": "example_emr_spark_streaming", "runtime": { "command": "EMR_SPARK_STREAMING" }, "content": "#!/bin/bash\necho 'hello'" } } ] } } ``` ## Reference - [EMR Spark Streaming Node](https://help.aliyun.com/zh/dataworks/user-guide/emr-spark-streaming-node) FILE:references/nodetypes/emr/EMR_STREAMING_SQL.md # EMR Streaming SQL（EMR_STREAMING_SQL） ## Overview - Compute engine: `EMR` - Content format: sql - Extension: `.sql` - Data source type: `emr` - Code: 266 - Description: Run Streaming SQL queries on EMR clusters Define and run stream processing tasks on EMR clusters using SQL, suitable for scenarios that describe stream data processing logic with SQL syntax, lowering the barrier to stream processing development. ## Prerequisites - An EMR cluster has been created and added as a DataWorks compute resource - The DataWorks resource group has network connectivity with the EMR cluster - EMR cluster has Spark service installed with Structured Streaming support ## Restrictions - Stream processing jobs are long-running tasks; ensure that the scheduling period and timeout are configured appropriately ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_emr_streaming_sql", "script": { "path": "example_emr_streaming_sql", "runtime": { "command": "EMR_STREAMING_SQL" }, "content": "SELECT 1;" } } ] } } ``` FILE:references/nodetypes/emr/EMR_TABLE.md # EMR Table（EMR_TABLE） ## Overview - Compute engine: `EMR` - Content format: empty (no code) - Extension: none - Data source type: `emr` - Code: 261 - LabelType：TABLE - Description: Manage Hive tables for EMR clusters in DataWorks Used in DataWorks to define and manage Hive/SparkSQL table structures on EMR clusters, supporting table creation, modification, and metadata management. ## Prerequisites - An EMR cluster has been created and added as a DataWorks compute resource - The DataWorks resource group has network connectivity with the EMR cluster - EMR cluster has Hive Metastore service installed ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_emr_table", "script": { "path": "example_emr_table", "runtime": { "command": "EMR_TABLE" }, "content": "" } } ] } } ``` FILE:references/nodetypes/emr/EMR_TRINO.md # EMR Trino SQL（EMR_TRINO） ## Overview - Compute engine: `EMR` - Content format: sql - Extension: `.sql` - Data source type: `emr` - Code: 267 - Description: Run Trino SQL queries on EMR clusters Execute SQL statements on the Trino engine of EMR clusters through DataWorks scheduling, suitable for interactive analysis and cross-data-source federated query scenarios. Trino (formerly PrestoSQL) is a high-performance distributed SQL query engine that supports federated queries across multiple data sources. ## Prerequisites - An EMR cluster has been created and added as a DataWorks compute resource - The DataWorks resource group has network connectivity with the EMR cluster - EMR cluster has Trino service installed ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_emr_trino", "script": { "path": "example_emr_trino", "runtime": { "command": "EMR_TRINO" }, "content": "SELECT 1;" } } ] } } ``` ## Reference - [EMR Trino Node](https://help.aliyun.com/zh/dataworks/user-guide/emr-trino-node) FILE:references/nodetypes/flink/BLINK_BATCH_SQL.md # Blink Batch SQL（BLINK_BATCH_SQL） ## Overview - Compute engine: `FLINK` - Content format: sql - Extension: `.json` - Data source type: `flink` - Code: 2020 - Description: Blink batch SQL (legacy), superseded by FLINK_SQL_BATCH > **Note**: BLINK_BATCH_SQL is a legacy node type. It is recommended to use [FLINK_SQL_BATCH](./FLINK_SQL_BATCH.md) instead. Do not use this node type for new projects. The Blink batch SQL node is an early batch SQL node based on the Alibaba Cloud Blink engine. Unlike BLINK_SQL (streaming), this node processes bounded datasets and executes in batch processing mode. The Blink engine has been unified and upgraded to the Flink edition, and this node type is retained only for compatibility with existing tasks. ## Prerequisites - Alibaba Cloud Realtime Compute service (Blink version) has been activated - Blink project association has been configured in the DataWorks workspace ## Core Features ### SQL Syntax Blink batch SQL syntax is used for batch processing of bounded datasets, supporting standard SELECT, JOIN, GROUP BY, and other operations. ```sql -- Blink Batch SQL example CREATE TABLE batch_input ( user_id BIGINT, item_id BIGINT, action VARCHAR ) WITH ( type = 'odps', tableName = 'user_behavior', partition = 'ds=20231001' ); CREATE TABLE batch_output ( user_id BIGINT, action_count BIGINT ) WITH ( type = 'odps', tableName = 'user_action_summary', partition = 'ds=20231001' ); INSERT INTO batch_output SELECT user_id, COUNT(*) AS action_count FROM batch_input GROUP BY user_id; ``` ## Migration Recommendation It is recommended to migrate existing BLINK_BATCH_SQL tasks to FLINK_SQL_BATCH. The main changes are: - Connector declaration: change `type = 'xxx'` to `'connector' = 'xxx'` - Compatibility adjustments for some data types and built-in functions ## Restrictions - Legacy node type; newer DataWorks versions may no longer support creation - Depends on the legacy Blink engine runtime environment - Batch tasks automatically end after processing completes ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_blink_batch_sql", "script": { "path": "example_blink_batch_sql", "runtime": { "command": "BLINK_BATCH_SQL" }, "content": "-- Blink Batch SQL\nSELECT 1;" } } ] } } ``` FILE:references/nodetypes/flink/BLINK_DATASTREAM.md # Blink DataStream（BLINK_DATASTREAM） ## Overview - Compute engine: `FLINK` - Content format: sql - Extension: `.json` - Data source type: `flink` - Code: 2019 - Description: Blink DataStream node (legacy), used to submit custom Flink DataStream jobs > **Note**: BLINK_DATASTREAM is a legacy node type. For new projects, it is recommended to use Flink SQL nodes (FLINK_SQL_STREAM / FLINK_SQL_BATCH) first, and only use the DataStream API when SQL cannot meet the requirements. The Blink DataStream node is used in DataWorks to submit custom Java/Scala jobs written with the Flink DataStream API. Unlike SQL nodes, DataStream nodes allow developers to implement more complex data processing logic through programming, such as custom windows, complex event processing (CEP), async I/O, and other advanced features. ## Prerequisites - Alibaba Cloud Realtime Compute service has been activated - Flink/Blink project association has been configured in the DataWorks workspace - The DataStream job has been packaged as a JAR file and uploaded to resource management ## Core Features ### Job Configuration The content of a DataStream node is a JSON configuration specifying the JAR package, main class, arguments, and other information: ```json { "jobType": "FLINK_DATASTREAM", "mainClass": "com.example.MyFlinkJob", "jarUri": "res:my_flink_job.jar", "args": "--input kafka_topic --output holo_table", "configuration": { "parallelism.default": "4", "taskmanager.memory.process.size": "2048m" } } ``` ### Applicable Scenarios - Complex business logic that cannot be expressed in SQL - Requires DataStream API advanced features (e.g., ProcessFunction, CEP) - Custom Source/Sink connector - Requires fine-grained control over state management and fault-tolerance mechanisms ## Restrictions - Legacy node type; newer DataWorks versions may no longer support creation - Java/Scala development skills are required - JAR packages must be uploaded to DataWorks resource management in advance - Debugging and troubleshooting are more difficult than SQL nodes ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_blink_datastream", "script": { "path": "example_blink_datastream", "runtime": { "command": "BLINK_DATASTREAM" }, "content": "SELECT 1;" } } ] } } ``` FILE:references/nodetypes/flink/BLINK_SQL.md # Blink Streaming SQL（BLINK_SQL） ## Overview - Compute engine: `FLINK` - Content format: sql - Extension: `.json` - Data source type: `flink` - Code: 2010 - Description: Blink streaming SQL (legacy), superseded by FLINK_SQL_STREAM > **Note**: BLINK_SQL is a legacy node type. It is recommended to use [FLINK_SQL_STREAM](./FLINK_SQL_STREAM.md) instead. Do not use this node type for new projects. The Blink streaming SQL node is an early real-time streaming computation node based on the Alibaba Cloud Blink engine. Blink is an Alibaba Cloud internal fork of Apache Flink, which was later unified and upgraded to the Flink edition for Alibaba Cloud's real-time computing service. This node type is retained only for compatibility with existing tasks. ## Prerequisites - Alibaba Cloud Realtime Compute service (Blink version) has been activated - Blink project association has been configured in the DataWorks workspace ## Core Features ### SQL Syntax Blink SQL syntax is largely consistent with Flink SQL, supporting DDL (CREATE TABLE), DML (INSERT INTO), and other operations. However, some syntax details and connector configurations differ from the newer Flink SQL version. ```sql -- Blink SQL example CREATE TABLE source_table ( id BIGINT, name VARCHAR, ts TIMESTAMP, WATERMARK FOR ts AS withOffset(ts, 1000) ) WITH ( type = 'kafka', topic = 'my_topic' ); CREATE TABLE sink_table ( id BIGINT, name VARCHAR ) WITH ( type = 'rds' ); INSERT INTO sink_table SELECT id, name FROM source_table; ``` ## Migration Recommendation It is recommended to migrate existing BLINK_SQL tasks to FLINK_SQL_STREAM. The main changes are: - Connector declaration: change `type = 'xxx'` to `'connector' = 'xxx'` - WATERMARK syntax adjustments - Some built-in function name changes ## Restrictions - Legacy node type; newer DataWorks versions may no longer support creation - Depends on the legacy Blink engine runtime environment ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_blink_sql", "script": { "path": "example_blink_sql", "runtime": { "command": "BLINK_STREAM_SQL" }, "content": "-- Blink SQL\nSELECT 1;" } } ] } } ``` FILE:references/nodetypes/flink/FLINK_SQL_BATCH.md # Flink Batch SQL（FLINK_SQL_BATCH） ## Overview - Compute engine: `FLINK` - Content format: sql - Extension: `.json` - Data source type: `flink` - Code: 2011 - Description: Flink batch SQL node, used for batch SQL computation based on the Flink engine The Flink batch SQL node is used in DataWorks to develop and schedule batch SQL tasks based on the Flink engine. Unlike streaming SQL, batch SQL processes bounded datasets and is suitable for offline batch computation scenarios. This node type leverages Flink's unified batch-streaming architecture to execute SQL queries in batch processing mode. ## Prerequisites - Alibaba Cloud Realtime Compute Flink edition has been activated and a Flink workspace has been created - A Flink compute engine instance has been bound in the DataWorks workspace - RAM users must have **Developer** or **Workspace Admin** role permissions ## Core Features ### SQL Syntax Flink batch SQL supports standard Flink SQL batch processing syntax, primarily used for: - Batch reading and processing of bounded data sources - Data aggregation, join queries, and other batch analysis operations - Batch data writes to target tables ```sql -- Define batch data source CREATE TABLE batch_source ( order_id BIGINT, user_id BIGINT, amount DECIMAL(10, 2), order_time TIMESTAMP(3) ) WITH ( 'connector' = 'filesystem', 'path' = 'oss://my-bucket/orders/', 'format' = 'parquet' ); -- Define target table CREATE TABLE batch_sink ( user_id BIGINT, total_amount DECIMAL(10, 2), order_count BIGINT ) WITH ( 'connector' = 'hologres', 'dbname' = 'my_db', 'tablename' = 'user_order_summary' ); -- Batch aggregation processing INSERT INTO batch_sink SELECT user_id, SUM(amount) AS total_amount, COUNT(*) AS order_count FROM batch_source GROUP BY user_id; ``` ### Scheduling Parameters Supports defining dynamic parameters using `${variable_name}`, which can be assigned values in the scheduling configuration. Time expressions such as `${yyyymmdd}` are also supported. ## Restrictions - Batch tasks automatically end after data processing completes, suitable for periodic scheduling - Although the content is in SQL format, it is actually stored as JSON configuration in DataWorks (extension `.json`) - The corresponding Flink compute resource and resource group must be selected in the run configuration ## Differences from Streaming SQL | Feature | Batch SQL (FLINK_SQL_BATCH) | Streaming SQL (FLINK_SQL_STREAM) | |------|---------------------------|---------------------------| | Dataset type | Bounded dataset | Unbounded data stream | | Execution method | Automatically ends after processing completes | Runs continuously | | Applicable scenarios | Offline batch analysis, ETL | Real-time data processing, monitoring | | Scheduling method | Periodic scheduling | Runs continuously after startup | ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_flink_sql_batch", "script": { "path": "example_flink_sql_batch", "runtime": { "command": "FLINK_SQL_BATCH" }, "content": "-- Flink Batch SQL\nSELECT 1;" } } ] } } ``` ## Related Documentation - [Flink Batch SQL Node](https://help.aliyun.com/zh/dataworks/user-guide/flink-sql-batch-node) FILE:references/nodetypes/flink/FLINK_SQL_STREAM.md # Flink Streaming SQL（FLINK_SQL_STREAM） ## Overview - Compute engine: `FLINK` - Content format: sql - Extension: `.json` - Data source type: `flink` - Code: 2012 - Description: Flink streaming SQL node, the recommended real-time computation node type The Flink streaming SQL node is used in DataWorks to develop and schedule Flink real-time computation tasks. Through standard Flink SQL syntax, you can define streaming data sources (Source), data targets (Sink), and data processing logic to achieve real-time data ingestion, transformation, and writing. This node type is the recommended approach for real-time stream computation in DataWorks, and has superseded the legacy BLINK_SQL. ## Prerequisites - Alibaba Cloud Realtime Compute Flink edition has been activated and a Flink workspace has been created - A Flink compute engine instance has been bound in the DataWorks workspace - RAM users must have **Developer** or **Workspace Admin** role permissions ## Core Features ### SQL Syntax Flink streaming SQL follows the standard Flink SQL syntax. A typical task consists of three parts: 1. **CREATE TABLE (Source)**: Define the source table, such as Kafka, DataHub, SLS, etc. 2. **CREATE TABLE (Sink)**: Define the target table, such as Hologres, MaxCompute, Kafka, etc. 3. **INSERT INTO ... SELECT**: Define the data transformation and write logic ```sql -- Define source table (Kafka) CREATE TABLE source_table ( user_id BIGINT, item_id BIGINT, behavior STRING, ts TIMESTAMP(3), WATERMARK FOR ts AS ts - INTERVAL '5' SECOND ) WITH ( 'connector' = 'kafka', 'topic' = 'user_behavior', 'properties.bootstrap.servers' = 'localhost:9092', 'format' = 'json' ); -- Define target table (Hologres) CREATE TABLE sink_table ( window_start TIMESTAMP(3), window_end TIMESTAMP(3), item_id BIGINT, pv_cnt BIGINT ) WITH ( 'connector' = 'hologres', 'dbname' = 'my_db', 'tablename' = 'item_pv', 'username' = '${ak}', 'password' = '${sk}' ); -- Data processing and write INSERT INTO sink_table SELECT TUMBLE_START(ts, INTERVAL '1' MINUTE) AS window_start, TUMBLE_END(ts, INTERVAL '1' MINUTE) AS window_end, item_id, COUNT(*) AS pv_cnt FROM source_table GROUP BY TUMBLE(ts, INTERVAL '1' MINUTE), item_id; ``` ### Scheduling Parameters Supports defining dynamic parameters using `${variable_name}`, which can be assigned values in the scheduling configuration. ## Restrictions - Once started, streaming tasks run continuously until manually stopped or an exception occurs - Although the content is in SQL format, it is actually stored as JSON configuration in DataWorks (extension `.json`) - The corresponding Flink compute resource and resource group must be selected in the run configuration ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_flink_sql_stream", "script": { "path": "example_flink_sql_stream", "runtime": { "command": "FLINK_SQL_STREAM" }, "content": "-- Flink Stream SQL\nSELECT 1;" } } ] } } ``` ## Related Documentation - [Flink Streaming SQL Node](https://help.aliyun.com/zh/dataworks/user-guide/flink-sql-streaming-node) FILE:references/nodetypes/general/CHECK.md # Check Node (CHECK) ## Overview - Compute engine: `GENERAL` - Content format: empty (no code) - Extension: `.json` - Description: Legacy check node for verifying the availability of target objects The Check node (legacy) is used to verify whether upstream data or target objects are ready before a workflow runs. Once the check conditions are met, the node returns a success status and triggers downstream task execution. For new tasks, it is recommended to use the new Check node (CHECK_NODE). ## Restrictions - This node has been replaced by the new CHECK_NODE; it is recommended to use the new version. - Check logic is configured via parameters; `script.content` is empty. ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_check", "script": { "path": "example_check", "runtime": { "command": "CHECK" }, "content": "" } } ] } } ``` FILE:references/nodetypes/general/CHECK_NODE.md # Check Node - New (CHECK_NODE) ## Overview - Compute engine: `GENERAL` - Content format: empty (no code) - Extension: `.json` - Description: New check node for verifying the availability of target objects The Check node is used to verify the availability of target objects. It supports checking MaxCompute partitioned tables, FTP files, OSS files, HDFS files, OSS_HDFS files, and Kafka-to-MaxCompute real-time sync tasks. Once the check conditions are met, the node returns a success status and triggers downstream task execution, ensuring the timing accuracy and automated execution of the workflow. ## Prerequisites - Data source-based check: The corresponding data source (MaxCompute, FTP, OSS, HDFS, or OSS_HDFS) must be created in advance. - Real-time sync task-based check: Only Kafka to MaxCompute tasks published to the production environment are supported. - Only supported on DataWorks Professional Edition or above. ## Restrictions - Only Serverless resource groups are supported for execution. - FTP data sources with Protocol configured as SFTP and key-based authentication are not supported. - A single check node can only check one object; multiple dependencies require multiple nodes. - The check interval range is 1-30 minutes. - Maximum running duration is 24 hours; the number of checks depends on the interval. - Available in limited regions: specific cities in East China, North China, South China, Southwest China, and Asia Pacific. - Scheduling resources are continuously occupied during node execution until the check completes. - Check logic is configured via parameters; `script.content` is empty. ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_check_node", "script": { "path": "example_check_node", "runtime": { "command": "CHECK_NODE" }, "content": "" } } ] } } ``` ## Reference - [Check Node](https://help.aliyun.com/zh/dataworks/user-guide/check-node) FILE:references/nodetypes/general/COMBINED_NODE.md # Combined Node (COMBINED_NODE) ## Overview - Compute engine: `GENERAL` - Content format: json - Extension: `.json` - Description: Combined node that encapsulates multiple logical steps into a single schedulable node unit The Combined node is used to encapsulate multiple ordered processing steps within a single node, executing them sequentially in order. It is suitable for scenarios where multiple related operations need to be scheduled and managed as a whole, simplifying the workflow structure. ## Content Structure `script.content` is the JSON configuration for the internal steps of the combined node. ## Restrictions - Internal steps of the combined node are executed serially in order. - If any internal step fails, the entire combined node is marked as failed. ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_combined_node", "script": { "path": "example_combined_node", "runtime": { "command": "COMBINED_NODE" }, "content": "{}" } } ] } } ``` ## Reference - [Combined Node](https://help.aliyun.com/zh/dataworks/user-guide/combined-node) FILE:references/nodetypes/general/CROSS_TENANTS.md # Cross-Tenant Node (CROSS_TENANTS) ## Overview - Compute engine: `GENERAL` - Content format: json - Extension: `.json` - Description: Cross-tenant node for task triggering and coordination across DataWorks tenants The Cross-Tenant node is used to establish task triggering relationships between different DataWorks tenants. By configuring the receive node identifier, it enables cross-tenant workflow coordination, suitable for data collaboration scenarios across multiple teams and organizations. ## Content Structure `script.content` is the JSON configuration for cross-tenant settings. The key fields are as follows: | Field | Type | Description | |------|------|------| | `receiveNodeIdentify` | string | Receive node identifier, specifying the receive node in the target tenant | ```json { "receiveNodeIdentify": "Node identifier in the target tenant" } ``` ## Restrictions - The target tenant must have authorized cross-tenant access for the current tenant. - The receive node identifier must exactly match the node in the target tenant. ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_cross_tenants", "script": { "path": "example_cross_tenants", "runtime": { "command": "CROSS_TENANTS" }, "content": "{\"receiveNodeIdentify\": \"example_cross_tenants\"}" } } ] } } ``` FILE:references/nodetypes/general/DATA_PUSH.md # Data Push (DATA_PUSH) ## Overview - Compute engine: `GENERAL` - Content format: json - Extension: `.json` - Description: Push data query results from upstream nodes to DingTalk groups, Feishu groups, WeCom groups, Teams, or email The Data Push node can push data query results from other nodes in a workflow to DingTalk groups, Feishu groups, WeCom groups, Teams, or email by configuring push targets. The node obtains upstream node outputs through context parameters and supports displaying data in Markdown or table format. ## Prerequisites - DataWorks service has been activated and a workspace has been created. - A Serverless resource group (must be created after June 28, 2024) is available. - The resource group has public network access enabled. ## Content Structure `script.content` is the JSON configuration for the push task. ## Restrictions - The upstream node must be a SQL query node or an assignment node. - Directly fetching data from ODPS SQL is not currently supported; an assignment node must be used as an intermediary. - Push content size limits: - DingTalk/Feishu: up to 20KB. - WeCom: 20 messages per bot per minute. - Teams: up to 28KB. - Email: Only one email body is supported. - Supported in the following regions only: China East 1 (Hangzhou), China East 2 (Shanghai), China North 2 (Beijing), and 9 other regions. ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_data_push", "script": { "path": "example_data_push", "runtime": { "command": "DATA_PUSH" }, "content": "{}" } } ] } } ``` ## Reference - [Data Push Node](https://help.aliyun.com/zh/dataworks/user-guide/data-push-node) FILE:references/nodetypes/general/DATA_QUALITY_MONITOR.md # Data Quality Monitor (DATA_QUALITY_MONITOR) ## Overview - Compute engine: `GENERAL` - Content format: json - Extension: `.json` - Description: Data quality monitor node that configures monitoring rules via JSON The Data Quality Monitor node is used to embed quality check rules into the data processing pipeline, performing monitoring checks on data tables or datasets. Monitoring rules can be configured to validate metrics such as data completeness and consistency, ensuring data quality meets expectations. ## Content Structure `script.content` is the JSON configuration for monitoring rules: ```json { "type": "Monitoring rule JSON configuration" } ``` ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_data_quality_monitor", "script": { "path": "example_data_quality_monitor", "runtime": { "command": "DATA_QUALITY_MONITOR" }, "content": "{}" } } ] } } ``` FILE:references/nodetypes/general/DATA_SYNCHRONIZATION_QUALITY_CHECK.md # Data Sync Quality Check (DATA_SYNCHRONIZATION_QUALITY_CHECK) ## Overview - Compute engine: `GENERAL` - Content format: json - Extension: `.json` - Description: Data sync quality check node that verifies data quality of sync tasks The Data Sync Quality Check node is used to perform quality verification on sync results after a data sync task completes. It can check the data consistency between the source and target, ensuring no data loss or anomalies occurred during the sync process. ## Content Structure `script.content` is the JSON configuration for quality checks: ```json { "type": "Quality check JSON configuration" } ``` ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_data_synchronization_quality_check", "script": { "path": "example_data_synchronization_quality_check", "runtime": { "command": "DATA_SYNCHRONIZATION_QUALITY_CHECK" }, "content": "{}" } } ] } } ``` FILE:references/nodetypes/general/DIDE_SHELL.md # Shell Script (DIDE_SHELL) ## Overview - Compute engine: `GENERAL` - Content format: shell - Extension: `.sh` - Description: Run standard Bash Shell scripts on DataWorks scheduling resource groups The Shell node supports standard Shell script execution, suitable for scenarios such as file operations, OSS/NAS data interaction, and batch data processing. The node has ossutil pre-installed for direct OSS storage operations, and supports resource references, scheduling parameter configuration, and RAM role association. ## Prerequisites - The RAM account must be added to the workspace with Developer or Workspace Admin role permissions. ## Restrictions - Supports standard Shell syntax; interactive syntax is not supported. - Serverless resource group supports a maximum of 64 CU per task; 16 CU or less is recommended. - Avoid launching too many subprocesses, as this may affect other tasks on the same resource group. - When calling other scripts, the Shell node must wait for the called script to complete. - Scheduling parameters only support positional parameter format (`$1`, `$2`, etc.); custom variable names are not supported. - Resources must be published before they can be referenced; the resource reference annotation `##@resource_reference{resource_name}` is a required identifier and must not be manually modified. - ossutil is pre-installed; no manual installation is required. ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_dide_shell", "script": { "path": "example_dide_shell", "runtime": { "command": "DIDE_SHELL" }, "content": "#!/bin/bash\necho \"Hello DataWorks\"\ndate" } } ] } } ``` ## Reference - [Shell Node](https://help.aliyun.com/zh/dataworks/user-guide/shell-node) FILE:references/nodetypes/general/FTP_CHECK.md # FTP File Check (FTP_CHECK) ## Overview - Compute engine: `GENERAL` - Content format: empty (no code) - Extension: `.json` - Description: FTP file check node that verifies the availability of files on FTP servers The FTP Check node is used to detect whether a specified file exists or is ready on an FTP server before workflow execution. Once the check conditions are met, the node returns a success status and triggers downstream task execution. ## Prerequisites - An FTP data source has been created, and network connectivity between the data source and the resource group is ensured. ## Restrictions - Check logic is configured via parameters; `script.content` is empty. - FTP data sources with Protocol configured as SFTP and key-based authentication are not supported. ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_ftp_check", "script": { "path": "example_ftp_check", "runtime": { "command": "FTP_CHECK" }, "content": "" } } ] } } ``` FILE:references/nodetypes/general/NOTEBOOK.md # Jupyter Notebook (NOTEBOOK) ## Overview - Compute engine: `GENERAL` - Content format: python - Extension: `.ipynb` - Description: Interactive, modular data analysis and development environment The Notebook node provides an interactive data analysis and development environment, supporting Python, SQL, and Markdown cells. It can connect to multiple compute engines such as MaxCompute, EMR, and AnalyticDB, enabling end-to-end tasks from data processing to model development. ## Prerequisites - The workspace has enabled the new Data Studio (Data Development). - A Serverless resource group has been created. - To use Python cells, a personal development environment instance must be created. ## Restrictions - Serverless resource groups recommend no more than 16 CU; maximum 64 CU per task. - Only nodes under the project directory can be published and periodically scheduled. - Lineage analysis only supports specific scenarios (between MaxCompute tables, and table-external data interactions). - Data Studio auto-save is not enabled by default; code must be saved manually. - Network policies differ between development and production environments; production dependencies must be ensured through custom images. - It is recommended to bind the personal development environment and resource group to the same VPC. ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_notebook", "script": { "path": "example_notebook", "runtime": { "command": "NOTEBOOK" }, "content": "" } } ] } } ``` ## Reference - [Notebook Node](https://help.aliyun.com/zh/dataworks/user-guide/notebook-node) FILE:references/nodetypes/general/OSS_INSPECT.md # OSS Object Inspection (OSS_INSPECT) ## Overview - Compute engine: `GENERAL` - Content format: empty (no code) - Extension: `.json` - Description: OSS object inspection node that verifies the availability of objects on OSS storage The OSS Inspect node is used to detect whether a specified object (file) exists or is ready on OSS storage before workflow execution. Once the check conditions are met, the node returns a success status and triggers downstream task execution, ensuring that the required OSS data is in place. ## Prerequisites - An OSS data source has been created, and network connectivity between the data source and the resource group is ensured. ## Restrictions - Check logic is configured via parameters; `script.content` is empty. - Only Serverless resource groups are supported for execution. ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_oss_inspect", "script": { "path": "example_oss_inspect", "runtime": { "command": "OSS_INSPECT" }, "content": "" } } ] } } ``` ## Reference - [OSS Sensor Node](https://help.aliyun.com/zh/dataworks/user-guide/oss-sensor-node) FILE:references/nodetypes/general/PERL.md # Perl Script (PERL) ## Overview - Compute engine: `GENERAL` - Content format: shell - Extension: `.pl` - Description: Run Perl scripts on DataWorks scheduling resource groups The Perl node is used for writing and scheduling Perl scripts, suitable for traditional Perl scripting scenarios such as text processing, log analysis, and system administration. ## Prerequisites - The RAM account must be added to the workspace with Developer or Workspace Admin role permissions. ## Restrictions - The runtime environment depends on the Perl interpreter version pre-installed on the scheduling resource group. - Interactive syntax is not supported. ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_perl", "script": { "path": "example_perl", "runtime": { "command": "PERL" }, "content": "#!/usr/bin/perl\nuse strict;\nuse warnings;\nprint \"Hello DataWorks\\n\";" } } ] } } ``` FILE:references/nodetypes/general/PYTHON.md # Python Script (PYTHON) ## Overview - Compute engine: `GENERAL` - Content format: python - Extension: `.py` - Description: Run Python 3 scripts on DataWorks scheduling resource groups The Python node is used for writing and scheduling Python 3 scripts, suitable for scenarios such as data processing, ETL helper logic, and calling external APIs. Scripts run in the Python 3 environment built into the scheduling resource group. ## Prerequisites - The RAM account must be added to the workspace with Developer or Workspace Admin role permissions. ## Restrictions - The runtime environment is Python 3; Python 2 syntax is not supported. - Serverless resource group supports a maximum of 64 CU per task; 16 CU or less is recommended. - Only pre-installed Python third-party libraries on the resource group can be used; additional dependencies must be uploaded via resource references. ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_python", "script": { "path": "example_python", "runtime": { "command": "PYTHON" }, "content": "import sys\nprint('Hello DataWorks')\nprint(f'Python version: {sys.version}')" } } ] } } ``` ## Reference - [Python Node](https://help.aliyun.com/zh/dataworks/user-guide/python-node) FILE:references/nodetypes/general/SSH.md # SSH Remote Execution (SSH) ## Overview - Compute engine: `GENERAL` - Content format: shell - Extension: `.ssh.sh` - Description: Remotely access hosts via SSH data source and execute scripts The SSH node allows DataWorks to remotely access hosts such as ECS through a specified SSH data source and trigger script execution on the remote host, enabling periodic scheduling of scripts. It is suitable for scenarios that require running tasks on remote servers. ## Prerequisites - An SSH data source has been created (only JDBC connection string method is supported). - Ensure the data source has network connectivity with the resource group. - The RAM account must have Developer or Workspace Admin role permissions. ## Restrictions - Only Serverless resource groups are supported for execution. - Supported in specific regions (East China, North China, South China, and some overseas regions). - Code length limit is 128KB. - Supports standard Shell syntax; interactive syntax is not supported. - When SSH tasks exit abnormally, operations on the underlying remote host are not affected. - Ensure sufficient ECS disk space for temporary file generation. - Avoid multiple tasks operating on the same file simultaneously to prevent node errors. ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_ssh", "script": { "path": "example_ssh", "runtime": { "command": "SSH" }, "content": "#!/bin/bash\necho \"Hello from SSH node\"\nhostname" } } ] } } ``` ## Reference - [SSH Node](https://help.aliyun.com/zh/dataworks/user-guide/ssh-node) FILE:references/nodetypes/general/SUB_PROCESS.md # Sub-workflow Container (SUB_PROCESS) ## Overview - Compute engine: `GENERAL` - Content format: empty (no code) - Extension: none - Description: Sub-workflow container node for nesting and referencing another workflow within a workflow The Sub-workflow container (also known as inner node) is used to embed a complete workflow as a sub-process within the current workflow. Sub-workflows enable modular management and reuse of processes, breaking down complex scheduling logic into multiple maintainable sub-processes. ## Restrictions - The sub-workflow container itself does not contain script content; `script.content` is empty. - Nodes within the sub-workflow are independently scheduled and executed according to their own dependency relationships. - The sub-workflow must be created and published before it can be referenced. ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_sub_process", "script": { "path": "example_sub_process", "runtime": { "command": "SUB_PROCESS" }, "content": "" } } ] } } ``` ## Reference - [Inner Node](https://help.aliyun.com/zh/dataworks/user-guide/inner-node) FILE:references/nodetypes/general/VIRTUAL.md # Virtual Node (VIRTUAL) ## Overview - Compute engine: `GENERAL` - Content format: empty (no code) - Extension: `.vi` - Description: An empty-run node that produces no data; the scheduler returns success directly The Virtual node is a control-type node. During scheduled execution, the system returns success directly without consuming resources, performing any operations, or blocking downstream execution. It is commonly used in the following scenarios: - As the orchestration starting node of a workflow, making the data flow path clearer. - As a consolidated output node for multiple branch nodes. - When multiple input nodes without dependency relationships need to be scheduled together, a Virtual node can serve as the upstream to manage downstream branches uniformly, and can control the earliest run time of each branch through scheduled timing. ## Prerequisites - The RAM account must be added to the corresponding workspace with Developer or Workspace Admin role permissions. - The workspace must have a Serverless resource group bound. ## Restrictions - The Virtual node has no script content; only scheduling properties need to be configured. - When the workspace root node is used as an upstream dependency, it is not displayed in the workflow panel and must be viewed in the Operation Center. ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_virtual", "script": { "path": "example_virtual", "runtime": { "command": "VIRTUAL" }, "content": "" } } ] } } ``` ## Reference - [Virtual Node](https://help.aliyun.com/zh/dataworks/user-guide/virtual-node) FILE:references/nodetypes/hologres/HOLOGRES_DEVELOP.md # Hologres Development（HOLOGRES_DEVELOP） ## Overview - Compute engine: `HOLO` - Content format: sql - Extension: `.sql` - Data source type: `hologres` - Code: 1091 - Description: Hologres development SQL node for Hologres database development and debugging The Hologres development node provides a SQL editing and debugging environment for the development phase, allowing you to write and run Hologres SQL directly in DataWorks. It is functionally similar to the HOLOGRES_SQL node, both using PostgreSQL-compatible syntax, and is suitable for table management, data querying, and ETL development in Hologres real-time data warehouses. ## Prerequisites - A Hologres compute engine instance has been added on the DataWorks workspace configuration page - RAM users must have **Developer** or **Workspace Admin** role permissions ## Core Features ### SQL Syntax Supports Hologres-compatible PostgreSQL syntax, including: - **DDL operations**: Create tables, modify table structures, create indexes, etc. - **DML operations**: INSERT, UPDATE, DELETE, SELECT - **External table queries**: Directly query MaxCompute data through external tables ```sql -- Create columnar storage table BEGIN; CREATE TABLE IF NOT EXISTS dwd_user_info ( user_id BIGINT NOT NULL, user_name TEXT, gender TEXT, register_date DATE, PRIMARY KEY (user_id) ); CALL set_table_property('dwd_user_info', 'orientation', 'column'); CALL set_table_property('dwd_user_info', 'bitmap_columns', 'gender'); COMMIT; -- Query using scheduling parameters SELECT user_id, user_name FROM dwd_user_info WHERE register_date = '${bizdate}'; ``` ### Scheduling Parameters Supports defining dynamic parameters using `${variable_name}` syntax, with values assigned in the scheduling configuration. ### Differences from HOLOGRES_SQL | Feature | HOLOGRES_DEVELOP | HOLOGRES_SQL | |------|------------------|-------------| | Code | 1091 | 1093 | | Purpose | Development and debugging | Production execution | | SQL syntax | PostgreSQL compatible | PostgreSQL compatible | The two are functionally identical; the main difference lies in node classification and intended use case. ## Restrictions - Query results display limit: up to 10,000 rows - Defaults to 200 rows when `LIMIT` is not specified ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_hologres_develop", "script": { "path": "example_hologres_develop", "runtime": { "command": "HOLOGRES_DEVELOP" }, "content": "SELECT 1;" } } ] } } ``` ## Related Documentation - [Hologres SQL Node](https://help.aliyun.com/zh/dataworks/user-guide/hologres-sql-node) FILE:references/nodetypes/hologres/HOLOGRES_SQL.md # Hologres SQL（HOLOGRES_SQL） ## Overview - Compute engine: `HOLO` - Content format: sql - Extension: `.sql` - Data source type: `hologres` - Code: 1093 - Description: Hologres SQL node for executing Hologres SQL queries and analysis Hologres is seamlessly integrated with MaxCompute at the underlying layer. Without migrating data, you can use the Hologres SQL node to directly query and analyze large-scale data in MaxCompute using standard PostgreSQL statements. This node supports Hologres-compatible PostgreSQL syntax and is suitable for data querying and analysis in real-time data warehouse scenarios. ## Prerequisites - A Hologres compute engine instance has been added on the DataWorks workspace configuration page - RAM users must have **Developer** or **Workspace Admin** role permissions ## Core Features ### SQL Syntax Hologres SQL is PostgreSQL-syntax compatible and supports standard DDL and DML operations: ```sql -- Create table CREATE TABLE IF NOT EXISTS user_profile ( user_id BIGINT NOT NULL, user_name TEXT, age INT, city TEXT, PRIMARY KEY (user_id) ); -- Query data (supports scheduling parameters) SELECT col_1, col_2 FROM your_table_name WHERE pt > ${pt_num} LIMIT 500; -- Joint query with MaxCompute external table SELECT h.user_id, h.user_name, m.order_amount FROM user_profile h JOIN odps_external_table m ON h.user_id = m.user_id WHERE m.dt = '${bizdate}'; ``` ### Scheduling Parameters Supports defining dynamic parameters using `${variable_name}` syntax, with values assigned in the scheduling configuration. ### Steps 1. Write task code in the SQL editing area 2. Select the compute resource and resource group in the run configuration 3. Select the created Hologres data source from the data source dropdown in the toolbar 4. Click Run to execute the task and save the node 5. Configure node scheduling information according to business requirements 6. Publish the node to the production environment ## Restrictions - Query results display limit: up to 10,000 rows - Defaults to 200 rows when `LIMIT` is not specified - Accessing data sources on public network or VPC environments requires using a scheduling resource group that has passed the connectivity test ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_hologres_sql", "script": { "path": "example_hologres_sql", "runtime": { "command": "HOLOGRES_SQL" }, "content": "SELECT 1;" } } ] } } ``` ## Related Documentation - [Hologres SQL Node](https://help.aliyun.com/zh/dataworks/user-guide/hologres-sql-node) FILE:references/nodetypes/hologres/HOLOGRES_SYNC.md # Hologres Data Sync（HOLOGRES_SYNC） ## Overview - Compute engine: `HOLO` - Content format: json - Extension: `.json` - Data source type: `hologres` - Code: 1092 - Description: Hologres data sync node for general Hologres data sync configuration The Hologres data sync node is used to configure Hologres-related data sync tasks. Through a JSON configuration file, it defines the data source, target, field mapping, and sync strategy to synchronize data between Hologres and other storage systems. ## Prerequisites - A Hologres compute engine instance has been added on the DataWorks workspace configuration page - Source and target data source connections required for data sync have been configured - RAM users must have **Developer** or **Workspace Admin** role permissions ## Core Features ### Configuration Structure The node content is a JSON sync configuration that defines the complete data sync workflow: ```json { "type": "HOLOGRES_SYNC", "reader": { "datasource": "source_ds_name", "table": "source_table", "columns": ["col1", "col2", "col3"] }, "writer": { "datasource": "hologres_ds_name", "table": "target_table", "columns": ["col1", "col2", "col3"], "writeMode": "INSERT_OR_REPLACE" }, "settings": { "speed": { "channel": 3 } } } ``` ### Applicable Scenarios - Syncing data from external data sources to Hologres - Syncing data between Hologres internal tables - General scenarios requiring custom sync strategies ## Restrictions - The sync configuration is in JSON format and must follow the specified schema - The sync task performance is limited by the resource configuration of the source and target ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_hologres_sync", "script": { "path": "example_hologres_sync", "runtime": { "command": "HOLOGRES_DEVELOP" }, "content": "{}" } } ] } } ``` FILE:references/nodetypes/hologres/HOLOGRES_SYNC_DATA.md # MaxCompute to Hologres Data Sync（HOLOGRES_SYNC_DATA） ## Overview - Compute engine: `HOLO` - Content format: json - Extension: `.hologres.data.sync.json` - Data source type: `hologres` - Code: 1095 - Description: MaxCompute to Hologres data sync node The HOLOGRES_SYNC_DATA node is used to sync data from MaxCompute (ODPS) to the Hologres real-time data warehouse. Through configuration, MaxCompute offline data can be efficiently imported into Hologres, bridging offline to real-time data. Both full and incremental sync modes are supported. ## Prerequisites - Hologres and MaxCompute compute engines have been bound in the DataWorks workspace - MaxCompute source tables and Hologres target tables have been created - MaxCompute and Hologres data source connections have been configured - RAM users must have **Developer** or **Workspace Admin** role permissions ## Core Features ### Configuration Structure The node content is in JSON format, defining the data sync configuration from MaxCompute to Hologres: ```json { "type": "HOLOGRES_SYNC_DATA", "source": { "datasource": "odps_datasource", "table": "mc_source_table", "partition": "ds=${bizdate}" }, "target": { "datasource": "hologres_datasource", "table": "holo_target_table", "writeMode": "INSERT_OR_REPLACE" }, "columnMapping": [ {"source": "user_id", "target": "user_id"}, {"source": "user_name", "target": "user_name"}, {"source": "amount", "target": "amount"} ] } ``` ### Sync Modes - **Full sync**: Imports all data from the specified MaxCompute partition into Hologres at once - **Incremental sync**: Combined with scheduling parameters (e.g., `${bizdate}`), incrementally syncs new data by partition - **Overwrite**: Clears the target table data before writing - **Append**: Appends new records on top of existing data ### Field Mapping Supports field mapping configuration between source and target tables, allowing field renaming and type conversion. ## Restrictions - MaxCompute and Hologres must be in the same region - It is recommended to configure concurrency appropriately when syncing large data volumes - The sync configuration is in JSON format, with extension `.hologres.data.sync.json` ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_hologres_sync_data", "script": { "path": "example_hologres_sync_data", "runtime": { "command": "HOLOGRES_SYNC_DATA" }, "content": "{}" } } ] } } ``` ## Related Documentation - [Sync Data from MaxCompute to Hologres](https://help.aliyun.com/zh/dataworks/user-guide/synchronize-data-from-maxcompute-to-hologres) FILE:references/nodetypes/hologres/HOLOGRES_SYNC_DATA_TO_MC.md # Hologres to MaxCompute Data Sync（HOLOGRES_SYNC_DATA_TO_MC） ## Overview - Compute engine: `HOLO` - Content format: json - Extension: `.hologres.data.sync.json` - Data source type: `hologres` - Code: 1070 - Description: Hologres to MaxCompute data sync node The HOLOGRES_SYNC_DATA_TO_MC node is used to reverse-sync data from the Hologres real-time data warehouse to MaxCompute. This node is suitable for archiving processed results from the real-time data warehouse to the offline data warehouse, or exporting aggregated data from Hologres to MaxCompute for further offline analysis and long-term storage. ## Prerequisites - Hologres and MaxCompute compute engines have been bound in the DataWorks workspace - Hologres source tables and MaxCompute target tables have been created - Hologres and MaxCompute data source connections have been configured - RAM users must have **Developer** or **Workspace Admin** role permissions ## Core Features ### Configuration Structure The node content is in JSON format, defining the data sync configuration from Hologres to MaxCompute: ```json { "type": "HOLOGRES_SYNC_DATA_TO_MC", "source": { "datasource": "hologres_datasource", "table": "holo_source_table" }, "target": { "datasource": "odps_datasource", "table": "mc_target_table", "partition": "ds=${bizdate}", "writeMode": "OVERWRITE" }, "columnMapping": [ {"source": "user_id", "target": "user_id"}, {"source": "total_amount", "target": "total_amount"}, {"source": "update_time", "target": "update_time"} ] } ``` ### Applicable Scenarios - **Data archival**: Archive hot data from Hologres to MaxCompute cold storage - **Offline analysis**: Export real-time processing results to MaxCompute for complex offline analysis - **Data backup**: Back up critical Hologres data to MaxCompute - **Cross-engine sharing**: Make Hologres data available to downstream tasks based on MaxCompute ### Write Modes - **OVERWRITE**: Overwrite data in the target partition - **APPEND**: Append data to the target partition ## Restrictions - Hologres and MaxCompute must be in the same region - When syncing large data volumes, pay attention to the impact on Hologres source query performance - The sync configuration is in JSON format, with extension `.hologres.data.sync.json` ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_hologres_sync_data_to_mc", "script": { "path": "example_hologres_sync_data_to_mc", "runtime": { "command": "HOLOGRES_SYNC_DATA_TO_MC" }, "content": "{}" } } ] } } ``` ## Related Documentation - [Sync Data from Hologres to MaxCompute](https://help.aliyun.com/zh/dataworks/user-guide/synchronize-data-from-hologres-to-maxcompute) FILE:references/nodetypes/hologres/HOLOGRES_SYNC_DDL.md # MaxCompute to Hologres Table Schema Sync（HOLOGRES_SYNC_DDL） ## Overview - Compute engine: `HOLO` - Content format: json - Extension: `.hologres.ddl.sync.json` - Data source type: `hologres` - Code: 1094 - Description: MaxCompute to Hologres table schema (DDL) sync node The HOLOGRES_SYNC_DDL node is used to automatically sync MaxCompute table schema definitions (DDL) to Hologres. This node reads the schema information of the MaxCompute source table (field names, field types, comments, etc.) and automatically creates or updates the corresponding table structure in Hologres, eliminating the need to manually recreate tables in Hologres. ## Prerequisites - Hologres and MaxCompute compute engines have been bound in the DataWorks workspace - MaxCompute source tables have been created - MaxCompute and Hologres data source connections have been configured - RAM users must have **Developer** or **Workspace Admin** role permissions ## Core Features ### Configuration Structure The node content is in JSON format, defining the source and target configuration for table schema sync: ```json { "type": "HOLOGRES_SYNC_DDL", "source": { "datasource": "odps_datasource", "table": "mc_source_table" }, "target": { "datasource": "hologres_datasource", "table": "holo_target_table", "schema": "public" }, "options": { "ifNotExists": true, "orientation": "column", "tableGroup": "default" } } ``` ### Sync Content The table schema sync includes the following elements: - **Field definitions**: Field names, data types (automatic type mapping) - **Field comments**: COMMENT information from MaxCompute fields - **Primary key constraints**: If the source table has primary key definitions, they are synced as well - **Partition information**: Handling of MaxCompute partition fields ### Type Mapping Common type mappings from MaxCompute to Hologres: | MaxCompute Type | Hologres Type | |----------------|--------------| | STRING | TEXT | | BIGINT | BIGINT | | INT | INT | | DOUBLE | DOUBLE PRECISION | | DECIMAL | NUMERIC | | BOOLEAN | BOOLEAN | | DATETIME | TIMESTAMPTZ | | DATE | DATE | ### Applicable Scenarios - **Unified table creation**: After defining table structures in MaxCompute, automatically sync them to Hologres - **Schema change sync**: Automatically update Hologres tables when MaxCompute table structures change - **Batch table creation**: Combined with scheduling, sync the schema of a large number of tables ## Restrictions - Only syncs table schema, not data (for data sync, use HOLOGRES_SYNC_DATA) - MaxCompute and Hologres must be in the same region - Some MaxCompute-specific data types may not be directly mappable to Hologres - The sync configuration is in JSON format, with extension `.hologres.ddl.sync.json` ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_hologres_sync_ddl", "script": { "path": "example_hologres_sync_ddl", "runtime": { "command": "HOLOGRES_SYNC_DDL" }, "content": "{}" } } ] } } ``` ## Related Documentation - [Sync Table Schema from MaxCompute to Hologres](https://help.aliyun.com/zh/dataworks/user-guide/synchronize-table-structures-from-maxcompute-to-hologres) FILE:references/nodetypes/index.md # Node Type Index Look up node type documentation by command name. The file name matches the command name; use Glob to locate directly: `**/ODPS_SQL.md` 130 node types in total. | Command | Description | Category | Engine | Format | Extension | Doc | |---------|-------------|----------|--------|--------|-----------|-----| | `ADB_SPARK` | ADB Spark | ADB Spark | `ADB_SPARK` | json | `.adb.spark.json` | [ADB_SPARK.md](adb_spark/ADB_SPARK.md) | | `ADB_SPARK_SQL` | ADB Spark SQL | ADB Spark | `ADB_SPARK` | sql | `.adb.spark.sql` | [ADB_SPARK_SQL.md](adb_spark/ADB_SPARK_SQL.md) | | `ALINK` | Alink | AI/Algorithm | `ALGORITHM` | - | `.alink.py` | [ALINK.md](ai/ALINK.md) | | `PAI` | PAI | AI/Algorithm | `ALGORITHM` | json | `.json` | [PAI.md](ai/PAI.md) | | `PAI_DLC` | PAI DLC | AI/Algorithm | `ALGORITHM` | - | `.pai.dlc.sh` | [PAI_DLC.md](ai/PAI_DLC.md) | | `PAI_FLOW` | PAI Flow | AI/Algorithm | `ALGORITHM` | yaml | `.yaml` | [PAI_FLOW.md](ai/PAI_FLOW.md) | | `PAI_STUDIO` | PAI Studio | AI/Algorithm | `ALGORITHM` | json | `.json` | [PAI_STUDIO.md](ai/PAI_STUDIO.md) | | `RECOMMEND_PLUS` | Recommendation Engine | AI/Algorithm | `ALGORITHM` | json | `.json` | [RECOMMEND_PLUS.md](ai/RECOMMEND_PLUS.md) | | `XLAB` | XLab | AI/Algorithm | `ALGORITHM` | - | `.xlab.json` | [XLAB.md](ai/XLAB.md) | | `CDH_FILE` | CDH File Resource | CDH | `HADOOP_CDH` | json | `.json` | [CDH_FILE.md](cdh/CDH_FILE.md) | | `CDH_FUNCTION` | CDH Function | CDH | `HADOOP_CDH` | - | - | [CDH_FUNCTION.md](cdh/CDH_FUNCTION.md) | | `CDH_HIVE` | CDH Hive | CDH | `HADOOP_CDH` | sql | `.sql` | [CDH_HIVE.md](cdh/CDH_HIVE.md) | | `CDH_IMPALA` | CDH Impala | CDH | `HADOOP_CDH` | sql | `.sql` | [CDH_IMPALA.md](cdh/CDH_IMPALA.md) | | `CDH_JAR` | CDH JAR Resource | CDH | `HADOOP_CDH` | json | `.json` | [CDH_JAR.md](cdh/CDH_JAR.md) | | `CDH_MR` | CDH MapReduce | CDH | `HADOOP_CDH` | - | - | [CDH_MR.md](cdh/CDH_MR.md) | | `CDH_PRESTO` | CDH Presto | CDH | `HADOOP_CDH` | sql | `.sql` | [CDH_PRESTO.md](cdh/CDH_PRESTO.md) | | `CDH_SHELL` | CDH Shell | CDH | `HADOOP_CDH` | shell | `.sh` | [CDH_SHELL.md](cdh/CDH_SHELL.md) | | `CDH_SPARK` | CDH Spark | CDH | `HADOOP_CDH` | shell | `.sh` | [CDH_SPARK.md](cdh/CDH_SPARK.md) | | `CDH_SPARK_SHELL` | CDH Spark Shell | CDH | `HADOOP_CDH` | shell | `.sh` | [CDH_SPARK_SHELL.md](cdh/CDH_SPARK_SHELL.md) | | `CDH_SPARK_SQL` | CDH Spark SQL | CDH | `HADOOP_CDH` | sql | `.sql` | [CDH_SPARK_SQL.md](cdh/CDH_SPARK_SQL.md) | | `CDH_TABLE` | CDH Table | CDH | `HADOOP_CDH` | - | - | [CDH_TABLE.md](cdh/CDH_TABLE.md) | | `CONTROLLER_ASSIGNMENT` | Assignment Node | Controller | `GENERAL` | json | `.assign.json` | [CONTROLLER_ASSIGNMENT.md](controller/CONTROLLER_ASSIGNMENT.md) | | `CONTROLLER_BRANCH` | Branch Node | Controller | `GENERAL` | - | `.branch.json` | [CONTROLLER_BRANCH.md](controller/CONTROLLER_BRANCH.md) | | `CONTROLLER_CYCLE` | Loop Node (do-while wrapper) | Controller | `GENERAL` | json | `.do-while.json` | [CONTROLLER_CYCLE.md](controller/CONTROLLER_CYCLE.md) | | `CONTROLLER_CYCLE_END` | Loop End Node | Controller | `GENERAL` | - | `.do-while-end` | [CONTROLLER_CYCLE_END.md](controller/CONTROLLER_CYCLE_END.md) | | `CONTROLLER_CYCLE_START` | Loop Start Node | Controller | `GENERAL` | - | `.do-while-start` | [CONTROLLER_CYCLE_START.md](controller/CONTROLLER_CYCLE_START.md) | | `CONTROLLER_JOIN` | Merge Node | Controller | `GENERAL` | json | `.join.json` | [CONTROLLER_JOIN.md](controller/CONTROLLER_JOIN.md) | | `CONTROLLER_TRAVERSE` | Traverse Node (for-each wrapper) | Controller | `GENERAL` | json | `.for-each.json` | [CONTROLLER_TRAVERSE.md](controller/CONTROLLER_TRAVERSE.md) | | `CONTROLLER_TRAVERSE_END` | Traverse End Node | Controller | `GENERAL` | - | `.for-each-end` | [CONTROLLER_TRAVERSE_END.md](controller/CONTROLLER_TRAVERSE_END.md) | | `CONTROLLER_TRAVERSE_START` | Traverse Start Node | Controller | `GENERAL` | - | `.for-each-start` | [CONTROLLER_TRAVERSE_START.md](controller/CONTROLLER_TRAVERSE_START.md) | | `PARAM_HUB` | Parameter Node | Controller | `GENERAL` | - | `.param-hub.json` | [PARAM_HUB.md](controller/PARAM_HUB.md) | | `SCHEDULER_TRIGGER` | Trigger Node | Controller | `GENERAL` | - | `.json` | [SCHEDULER_TRIGGER.md](controller/SCHEDULER_TRIGGER.md) | | `CUSTOM` | Custom Node | Custom | `CUSTOM` | json | `.json` | [CUSTOM.md](custom/CUSTOM.md) | | `DATAX` | DataX | Data Integration | `DI` | json | `.json` | [DATAX.md](data_integration/DATAX.md) | | `DATAX2` | DataX2 | Data Integration | `DI` | json | `.json` | [DATAX2.md](data_integration/DATAX2.md) | | `DD_MERGE` | Data Merge | Data Integration | `DI` | json | `.json` | [DD_MERGE.md](data_integration/DD_MERGE.md) | | `DI` | Data Integration (Offline Sync) | Data Integration | `DI` | json | `.json` | [DI.md](data_integration/DI.md) | | `DT` | DT Sync | Data Integration | `DI` | json | `.json` | [DT.md](data_integration/DT.md) | | `RI` | Real-time Integration | Data Integration | `DI` | json | `.json` | [RI.md](data_integration/RI.md) | | `TT_MERGE` | Table Merge | Data Integration | `DI` | json | `.json` | [TT_MERGE.md](data_integration/TT_MERGE.md) | | `ADB_for_MySQL` | AnalyticDB for MySQL | Database SQL | `DATABASE` | sql | `.sql` | [ADB_for_MySQL.md](database/ADB_for_MySQL.md) | | `ADB_for_PostgreSQL` | AnalyticDB for PostgreSQL | Database SQL | `DATABASE` | sql | `.sql` | [ADB_for_PostgreSQL.md](database/ADB_for_PostgreSQL.md) | | `CLICK_SQL` | ClickHouse SQL | Database SQL | `CLICKHOUSE` | sql | `.sql` | [CLICK_SQL.md](database/CLICK_SQL.md) | | `DB2` | DB2 | Database SQL | `DATABASE` | sql | `.sql` | [DB2.md](database/DB2.md) | | `DRDS` | DRDS | Database SQL | `DATABASE` | sql | `.sql` | [DRDS.md](database/DRDS.md) | | `Doris` | Doris | Database SQL | `DATABASE` | sql | `.sql` | [Doris.md](database/Doris.md) | | `MYSQL` | MySQL | Database SQL | `DATABASE` | sql | `.sql` | [MYSQL.md](database/MYSQL.md) | | `Mariadb` | MariaDB | Database SQL | `DATABASE` | sql | `.sql` | [Mariadb.md](database/Mariadb.md) | | `OceanBase` | OceanBase | Database SQL | `DATABASE` | sql | `.sql` | [OceanBase.md](database/OceanBase.md) | | `Oracle` | Oracle | Database SQL | `DATABASE` | sql | `.sql` | [Oracle.md](database/Oracle.md) | | `POSTGRESQL` | PostgreSQL | Database SQL | `DATABASE` | sql | `.sql` | [POSTGRESQL.md](database/POSTGRESQL.md) | | `Redshift` | Redshift | Database SQL | `DATABASE` | sql | `.sql` | [Redshift.md](database/Redshift.md) | | `SQLSERVER` | SQL Server | Database SQL | `DATABASE` | sql | `.sql` | [SQLSERVER.md](database/SQLSERVER.md) | | `Saphana` | SAP HANA | Database SQL | `DATABASE` | sql | `.sql` | [Saphana.md](database/Saphana.md) | | `Selectdb` | SelectDB | Database SQL | `DATABASE` | sql | `.sql` | [Selectdb.md](database/Selectdb.md) | | `StarRocks` | StarRocks | Database SQL | `DATABASE` | sql | `.sql` | [StarRocks.md](database/StarRocks.md) | | `Vertica` | Vertica | Database SQL | `DATABASE` | sql | `.sql` | [Vertica.md](database/Vertica.md) | | `EMR_FILE` | EMR File Resource | EMR | `EMR` | json | `.json` | [EMR_FILE.md](emr/EMR_FILE.md) | | `EMR_FUNCTION` | EMR Function | EMR | `EMR` | - | - | [EMR_FUNCTION.md](emr/EMR_FUNCTION.md) | | `EMR_HIVE` | EMR Hive | EMR | `EMR` | sql | `.sql` | [EMR_HIVE.md](emr/EMR_HIVE.md) | | `EMR_HIVE_CLI` | EMR Hive CLI | EMR | `EMR` | - | - | [EMR_HIVE_CLI.md](emr/EMR_HIVE_CLI.md) | | `EMR_IMPALA` | EMR Impala | EMR | `EMR` | sql | `.sql` | [EMR_IMPALA.md](emr/EMR_IMPALA.md) | | `EMR_JAR` | EMR JAR Resource | EMR | `EMR` | json | `.json` | [EMR_JAR.md](emr/EMR_JAR.md) | | `EMR_KYUUBI` | EMR Kyuubi | EMR | `EMR` | sql | `.sql` | [EMR_KYUUBI.md](emr/EMR_KYUUBI.md) | | `EMR_MR` | EMR MapReduce | EMR | `EMR` | shell | `.sh` | [EMR_MR.md](emr/EMR_MR.md) | | `EMR_PRESTO` | EMR Presto | EMR | `EMR` | sql | `.sql` | [EMR_PRESTO.md](emr/EMR_PRESTO.md) | | `EMR_PYSPARK` | EMR PySpark | EMR | `EMR` | python | `.py` | [EMR_PYSPARK.md](emr/EMR_PYSPARK.md) | | `EMR_SCOOP` | EMR Sqoop | EMR | `EMR` | - | - | [EMR_SCOOP.md](emr/EMR_SCOOP.md) | | `EMR_SHELL` | EMR Shell | EMR | `EMR` | shell | `.sh` | [EMR_SHELL.md](emr/EMR_SHELL.md) | | `EMR_SPARK` | EMR Spark | EMR | `EMR` | shell | `.sh` | [EMR_SPARK.md](emr/EMR_SPARK.md) | | `EMR_SPARK_SHELL` | EMR Spark Shell | EMR | `EMR` | shell | `.sh` | [EMR_SPARK_SHELL.md](emr/EMR_SPARK_SHELL.md) | | `EMR_SPARK_SQL` | EMR Spark SQL | EMR | `EMR` | sql | `.sql` | [EMR_SPARK_SQL.md](emr/EMR_SPARK_SQL.md) | | `EMR_SPARK_STREAMING` | EMR Spark Streaming | EMR | `EMR` | shell | `.sh` | [EMR_SPARK_STREAMING.md](emr/EMR_SPARK_STREAMING.md) | | `EMR_STREAMING_SQL` | EMR Streaming SQL | EMR | `EMR` | sql | `.sql` | [EMR_STREAMING_SQL.md](emr/EMR_STREAMING_SQL.md) | | `EMR_TABLE` | EMR Table | EMR | `EMR` | - | - | [EMR_TABLE.md](emr/EMR_TABLE.md) | | `EMR_TRINO` | EMR Trino | EMR | `EMR` | sql | `.sql` | [EMR_TRINO.md](emr/EMR_TRINO.md) | | `BLINK_BATCH_SQL` | Blink Batch SQL | Flink | `FLINK` | sql | `.json` | [BLINK_BATCH_SQL.md](flink/BLINK_BATCH_SQL.md) | | `BLINK_DATASTREAM` | Blink DataStream | Flink | `FLINK` | sql | `.json` | [BLINK_DATASTREAM.md](flink/BLINK_DATASTREAM.md) | | `BLINK_SQL` | Blink Streaming SQL | Flink | `FLINK` | sql | `.json` | [BLINK_SQL.md](flink/BLINK_SQL.md) | | `FLINK_SQL_BATCH` | Flink Batch SQL | Flink | `FLINK` | sql | `.json` | [FLINK_SQL_BATCH.md](flink/FLINK_SQL_BATCH.md) | | `FLINK_SQL_STREAM` | Flink Streaming SQL | Flink | `FLINK` | sql | `.json` | [FLINK_SQL_STREAM.md](flink/FLINK_SQL_STREAM.md) | | `CHECK` | Check Node | General | `GENERAL` | - | `.json` | [CHECK.md](general/CHECK.md) | | `CHECK_NODE` | Check Node (New) | General | `GENERAL` | - | `.json` | [CHECK_NODE.md](general/CHECK_NODE.md) | | `COMBINED_NODE` | Combined Node | General | `GENERAL` | json | `.json` | [COMBINED_NODE.md](general/COMBINED_NODE.md) | | `CROSS_TENANTS` | Cross-Tenant Node | General | `GENERAL` | json | `.json` | [CROSS_TENANTS.md](general/CROSS_TENANTS.md) | | `DATA_PUSH` | Data Push | General | `GENERAL` | json | `.json` | [DATA_PUSH.md](general/DATA_PUSH.md) | | `DATA_QUALITY_MONITOR` | Data Quality Monitor | General | `GENERAL` | json | `.json` | [DATA_QUALITY_MONITOR.md](general/DATA_QUALITY_MONITOR.md) | | `DATA_SYNCHRONIZATION_QUALITY_CHECK` | Data Sync Quality Check | General | `GENERAL` | json | `.json` | [DATA_SYNCHRONIZATION_QUALITY_CHECK.md](general/DATA_SYNCHRONIZATION_QUALITY_CHECK.md) | | `DIDE_SHELL` | Shell Script | General | `GENERAL` | shell | `.sh` | [DIDE_SHELL.md](general/DIDE_SHELL.md) | | `FTP_CHECK` | FTP Check | General | `GENERAL` | - | `.json` | [FTP_CHECK.md](general/FTP_CHECK.md) | | `NOTEBOOK` | Notebook | General | `GENERAL` | python | `.ipynb` | [NOTEBOOK.md](general/NOTEBOOK.md) | | `OSS_INSPECT` | OSS Inspect | General | `GENERAL` | - | `.json` | [OSS_INSPECT.md](general/OSS_INSPECT.md) | | `PERL` | Perl Script | General | `GENERAL` | shell | `.pl` | [PERL.md](general/PERL.md) | | `PYTHON` | Python Script | General | `GENERAL` | python | `.py` | [PYTHON.md](general/PYTHON.md) | | `SSH` | SSH Script | General | `GENERAL` | shell | `.ssh.sh` | [SSH.md](general/SSH.md) | | `SUB_PROCESS` | Sub-process | General | `GENERAL` | - | - | [SUB_PROCESS.md](general/SUB_PROCESS.md) | | `VIRTUAL` | Virtual Node | General | `GENERAL` | - | `.vi` | [VIRTUAL.md](general/VIRTUAL.md) | | `HOLOGRES_DEVELOP` | Hologres Development | Hologres | `HOLO` | sql | `.sql` | [HOLOGRES_DEVELOP.md](hologres/HOLOGRES_DEVELOP.md) | | `HOLOGRES_SQL` | Hologres SQL | Hologres | `HOLO` | sql | `.sql` | [HOLOGRES_SQL.md](hologres/HOLOGRES_SQL.md) | | `HOLOGRES_SYNC` | Hologres Sync | Hologres | `HOLO` | json | `.json` | [HOLOGRES_SYNC.md](hologres/HOLOGRES_SYNC.md) | | `HOLOGRES_SYNC_DATA` | Hologres Data Sync | Hologres | `HOLO` | json | `.hologres.data.sync.json` | [HOLOGRES_SYNC_DATA.md](hologres/HOLOGRES_SYNC_DATA.md) | | `HOLOGRES_SYNC_DATA_TO_MC` | Hologres Data Sync to MaxCompute | Hologres | `HOLO` | json | `.hologres.data.sync.json` | [HOLOGRES_SYNC_DATA_TO_MC.md](hologres/HOLOGRES_SYNC_DATA_TO_MC.md) | | `HOLOGRES_SYNC_DDL` | Hologres DDL Sync | Hologres | `HOLO` | json | `.hologres.ddl.sync.json` | [HOLOGRES_SYNC_DDL.md](hologres/HOLOGRES_SYNC_DDL.md) | | `COMPONENT_SQL` | SQL Component | MaxCompute | `ODPS` | sql | `.sql` | [COMPONENT_SQL.md](maxcompute/COMPONENT_SQL.md) | | `DATASERVICE_STUDIO` | Data Service | MaxCompute | `ODPS` | sql | `.json` | [DATASERVICE_STUDIO.md](maxcompute/DATASERVICE_STUDIO.md) | | `EXTREME_STORAGE` | MaxCompute Extreme Storage | MaxCompute | `ODPS` | - | `.mc.extreme.store.sh` | [EXTREME_STORAGE.md](maxcompute/EXTREME_STORAGE.md) | | `LIGHTNING_SQL` | Lightning SQL | MaxCompute | `ODPS` | sql | `.sql` | [LIGHTNING_SQL.md](maxcompute/LIGHTNING_SQL.md) | | `ODPS_ARCHIVE` | MaxCompute Archive Resource | MaxCompute | `ODPS` | - | `.json` | [ODPS_ARCHIVE.md](maxcompute/ODPS_ARCHIVE.md) | | `ODPS_DDL` | MaxCompute DDL | MaxCompute | `ODPS` | sql | `.json` | [ODPS_DDL.md](maxcompute/ODPS_DDL.md) | | `ODPS_FILE` | MaxCompute File Resource | MaxCompute | `ODPS` | - | `.json` | [ODPS_FILE.md](maxcompute/ODPS_FILE.md) | | `ODPS_FUNCTION` | MaxCompute Function | MaxCompute | `ODPS` | json | `.json` | [ODPS_FUNCTION.md](maxcompute/ODPS_FUNCTION.md) | | `ODPS_JAR` | MaxCompute JAR Resource | MaxCompute | `ODPS` | - | `.json` | [ODPS_JAR.md](maxcompute/ODPS_JAR.md) | | `ODPS_MR` | MaxCompute MapReduce | MaxCompute | `ODPS` | sql | `.mr.sql` | [ODPS_MR.md](maxcompute/ODPS_MR.md) | | `ODPS_PERL` | MaxCompute Perl | MaxCompute | `ODPS` | shell | `.mc.pl` | [ODPS_PERL.md](maxcompute/ODPS_PERL.md) | | `ODPS_PYTHON` | MaxCompute Python Resource | MaxCompute | `ODPS` | - | `.json` | [ODPS_PYTHON.md](maxcompute/ODPS_PYTHON.md) | | `ODPS_SCRIPT` | MaxCompute Script | MaxCompute | `ODPS` | sql | `.ms` | [ODPS_SCRIPT.md](maxcompute/ODPS_SCRIPT.md) | | `ODPS_SHARK` | MaxCompute Shark | MaxCompute | `ODPS` | json | `.mc.shark.json` | [ODPS_SHARK.md](maxcompute/ODPS_SHARK.md) | | `ODPS_SPARK` | MaxCompute Spark | MaxCompute | `ODPS` | json | `.mc.spark.json` | [ODPS_SPARK.md](maxcompute/ODPS_SPARK.md) | | `ODPS_SQL` | MaxCompute SQL | MaxCompute | `ODPS` | sql | `.sql` | [ODPS_SQL.md](maxcompute/ODPS_SQL.md) | | `ODPS_TABLE` | MaxCompute Table | MaxCompute | `ODPS` | json | `.json` | [ODPS_TABLE.md](maxcompute/ODPS_TABLE.md) | | `ODPS_XLIB` | MaxCompute XLib | MaxCompute | `ODPS` | python | `.mc.xlib.py` | [ODPS_XLIB.md](maxcompute/ODPS_XLIB.md) | | `PYODPS` | PyODPS 2 | MaxCompute | `ODPS` | python | `.py` | [PYODPS.md](maxcompute/PYODPS.md) | | `PYODPS3` | PyODPS 3 | MaxCompute | `ODPS` | python | `.py` | [PYODPS3.md](maxcompute/PYODPS3.md) | | `SQL_COMPONENT` | SQL Component (New) | MaxCompute | `ODPS` | sql | `.sql` | [SQL_COMPONENT.md](maxcompute/SQL_COMPONENT.md) | | `YSF_DESEN` | MaxCompute Data Masking | MaxCompute | `ODPS` | sql | `.mc.data.masking.sql` | [YSF_DESEN.md](maxcompute/YSF_DESEN.md) | | `SERVERLESS_KYUUBI` | Serverless Kyuubi | Serverless Spark | `EMR` | sql | `.sql` | [SERVERLESS_KYUUBI.md](serverless_spark/SERVERLESS_KYUUBI.md) | | `SERVERLESS_PYSPARK` | Serverless PySpark | Serverless Spark | `EMR` | python | `.py` | [SERVERLESS_PYSPARK.md](serverless_spark/SERVERLESS_PYSPARK.md) | | `SERVERLESS_SPARK_BATCH` | Serverless Spark Batch | Serverless Spark | `EMR` | shell | `.sh` | [SERVERLESS_SPARK_BATCH.md](serverless_spark/SERVERLESS_SPARK_BATCH.md) | | `SERVERLESS_SPARK_SQL` | Serverless Spark SQL | Serverless Spark | `EMR` | sql | `.sql` | [SERVERLESS_SPARK_SQL.md](serverless_spark/SERVERLESS_SPARK_SQL.md) | | `SERVERLESS_SPARK_STREAMING` | Serverless Spark Streaming | Serverless Spark | `EMR` | shell | `.sh` | [SERVERLESS_SPARK_STREAMING.md](serverless_spark/SERVERLESS_SPARK_STREAMING.md) | FILE:references/nodetypes/maxcompute/COMPONENT_SQL.md # SQL Component（COMPONENT_SQL） ## Overview - Compute engine: `ODPS` - Content format: sql - Extension: `.sql` - Code: 1010 - Data source type: `odps` - Description: SQL component (legacy), supports parameterized SQL > **Note**: COMPONENT_SQL is the legacy SQL component. It is recommended to use [SQL_COMPONENT](./SQL_COMPONENT.md) (new version) instead. The SQL component node provides SQL code templates with multiple input and output parameters. It generates result tables by performing filtering, joining, and aggregation operations on data source tables, supporting parameterized SQL reuse. ## Prerequisites - The workspace has been bound to a MaxCompute compute resource ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_component_sql", "script": { "path": "example_component_sql", "runtime": { "command": "COMPONENT_SQL" }, "content": "SELECT ${param1} FROM dual;" } } ] } } ``` FILE:references/nodetypes/maxcompute/DATASERVICE_STUDIO.md # Data Service（DATASERVICE_STUDIO） ## Overview - Compute engine: `ODPS` - Content format: sql - Extension: `.json` - Code: 238 - Data source type: `odps` - Description: Data Service SQL The Data Service node is used to define API query logic in DataWorks Data Service, exposing MaxCompute data as APIs through SQL configuration. ## Prerequisites - The workspace has been bound to a MaxCompute compute resource - DataWorks Data Service feature has been activated ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_dataservice_studio", "script": { "path": "example_dataservice_studio", "runtime": { "command": "DataService_studio" }, "content": "{}" } } ] } } ``` FILE:references/nodetypes/maxcompute/EXTREME_STORAGE.md # MaxCompute Extreme Storage（EXTREME_STORAGE） ## Overview - Compute engine: `ODPS` - Content format: empty (no code) - Extension: `.mc.extreme.store.sh` - Code: 30 - Data source type: `odps` - Description: Extreme storage node The extreme storage node is used to manage MaxCompute table storage format conversion, converting table data to the extreme storage format to improve query performance. This node does not require writing code and can be completed through configuration. ## Prerequisites - The workspace has been bound to a MaxCompute compute resource ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_extreme_storage", "script": { "path": "example_extreme_storage", "runtime": { "command": "EXTREME_STORAGE" }, "content": "" } } ] } } ``` FILE:references/nodetypes/maxcompute/LIGHTNING_SQL.md # Lightning SQL（LIGHTNING_SQL） ## Overview - Compute engine: `ODPS` - Content format: sql - Extension: `.sql` - Code: 61 - Data source type: `odps` - Description: Lightning SQL query The Lightning SQL node is used to perform interactive SQL queries through the MaxCompute Lightning service. Lightning provides a PostgreSQL-compatible query interface that supports low-latency interactive analysis on MaxCompute tables. ## Prerequisites - The workspace has been bound to a MaxCompute compute resource - MaxCompute Lightning service has been activated ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_lightning_sql", "script": { "path": "example_lightning_sql", "runtime": { "command": "LIGHTNING_SQL" }, "content": "SELECT 1;" } } ] } } ``` FILE:references/nodetypes/maxcompute/ODPS_ARCHIVE.md # MaxCompute Archive Resource（ODPS_ARCHIVE） ## Overview - Compute engine: `ODPS` - Content format: empty (no code) - Extension: `.json` - Code: 14 - Data source type: `odps` - Label type: RESOURCE - Description: MaxCompute Archive resource The Archive resource node is used to upload compressed archive files (such as .tar.gz, .zip, etc.) to a MaxCompute project for use by MapReduce jobs, Spark jobs, etc. Resources must be published after upload before they can be used by other nodes. ## Prerequisites - The workspace has been bound to a MaxCompute compute resource ## Usage Notes - Resources must be published before they can be referenced by other nodes - Supported archive formats include .tar.gz, .zip, etc. - Commonly used to upload dependency packages containing multiple files ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_odps_archive", "script": { "path": "example_odps_archive", "runtime": { "command": "ODPS_ARCHIVE" }, "content": "" } } ] } } ``` FILE:references/nodetypes/maxcompute/ODPS_DDL.md # MaxCompute DDL（ODPS_DDL） ## Overview - Compute engine: `ODPS` - Content format: sql - Extension: `.json` - Code: 18 - Data source type: `odps` - Label type: RESOURCE - Description: MaxCompute DDL statements (CREATE/ALTER/DROP) The DDL node is used to execute MaxCompute Data Definition Language (DDL) operations, including creating tables, altering table structures, dropping tables, etc. It is suitable for scenarios that require automated table structure management through scheduling. ## Prerequisites - The workspace has been bound to a MaxCompute compute resource ## Core Features Supports standard MaxCompute DDL statements: ```sql -- Create table CREATE TABLE IF NOT EXISTS my_table ( id BIGINT, name STRING, amount DOUBLE ) PARTITIONED BY (dt STRING) LIFECYCLE 180; -- Alter table ALTER TABLE my_table ADD COLUMNS (city STRING); -- Drop table DROP TABLE IF EXISTS temp_table; ``` ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_odps_ddl", "script": { "path": "example_odps_ddl", "runtime": { "command": "ODPS_DDL" }, "content": "CREATE TABLE IF NOT EXISTS test_table (id BIGINT, name STRING);" } } ] } } ``` FILE:references/nodetypes/maxcompute/ODPS_FILE.md # MaxCompute File Resource（ODPS_FILE） ## Overview - Compute engine: `ODPS` - Content format: empty (no code) - Extension: `.json` - Code: 15 - Data source type: `odps` - Label type: RESOURCE - Description: MaxCompute file resource The file resource node is used to upload text files, configuration files, etc. to a MaxCompute project for use by SQL nodes, MapReduce jobs, etc. Resources must be published after upload before they can be used by other nodes. ## Prerequisites - The workspace has been bound to a MaxCompute compute resource ## Usage Notes - Resources must be published before they can be referenced by other nodes - Commonly used to upload configuration files, dictionary files, etc. - Can be referenced in SQL nodes via the `add file` command ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_odps_file", "script": { "path": "example_odps_file", "runtime": { "command": "ODPS_FILE" }, "content": "" } } ] } } ``` FILE:references/nodetypes/maxcompute/ODPS_FUNCTION.md # MaxCompute Function（ODPS_FUNCTION） ## Overview - Compute engine: `ODPS` - Content format: json - Extension: `.json` - Code: 17 - Data source type: `odps` - Label type: FUNCTION - Description: MaxCompute UDF function definition The MaxCompute function node is used to define UDF (User Defined Function) functions in JSON format, specifying the function name, Java class name, and dependent resource files. This node enables version management and automated deployment of custom functions. ## Prerequisites - The workspace has been bound to a MaxCompute compute resource - The JAR resources that the function depends on have been uploaded and published (via the ODPS_JAR node) ## Content Structure ```json { "name": "function name", "className": "Java fully qualified class name", "resources": ["dependent resource file names"] } ``` ### UDF Types MaxCompute supports three types of custom functions: - **UDF**: Takes one row as input and outputs one row (e.g., format conversion, string processing) - **UDAF**: Takes multiple rows as input and outputs one row (e.g., custom aggregate functions) - **UDTF**: Takes one row as input and outputs multiple rows (e.g., data splitting, expanding) ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_odps_function", "script": { "path": "example_odps_function", "runtime": { "command": "ODPS_FUNCTION" }, "content": "{\"name\":\"example_func\",\"className\":\"com.example.UDF\",\"resources\":[\"example.jar\"]}" } } ] } } ``` FILE:references/nodetypes/maxcompute/ODPS_JAR.md # MaxCompute JAR Resource（ODPS_JAR） ## Overview - Compute engine: `ODPS` - Content format: empty (no code) - Extension: `.json` - Code: 13 - Data source type: `odps` - Label type: RESOURCE - Description: MaxCompute JAR resource The JAR resource node is used to upload Java JAR packages to a MaxCompute project for use by MapReduce jobs, UDF functions, Spark jobs, etc. Resources must be published after upload before they can be used by other nodes. ## Prerequisites - The workspace has been bound to a MaxCompute compute resource ## Usage Notes - Resources must be published before they can be referenced by other nodes - Referenced by resource name in ODPS_MR or ODPS_SPARK nodes - Used as a UDF dependency resource in ODPS_FUNCTION ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_odps_jar", "script": { "path": "example_odps_jar", "runtime": { "command": "ODPS_JAR" }, "content": "" } } ] } } ``` FILE:references/nodetypes/maxcompute/ODPS_MR.md # MaxCompute MapReduce（ODPS_MR） ## Overview - Compute engine: `ODPS` - Content format: sql - Extension: `.mr.sql` - Code: 11 - Data source type: `odps` - Description: MaxCompute MapReduce command The MaxCompute MR node is used to process large-scale datasets in MaxCompute through programs written with the MapReduce Java API. The node content is a jar command that specifies the JAR resource and main class to execute. ## Prerequisites - The workspace has been bound to a MaxCompute compute resource - The required JAR packages and resource files have been uploaded and published in advance ## Core Features ### Command Format ```sql jar -resources -classpath ./

; ``` Multiple JAR resources are separated by commas: ```sql jar -resources example1.jar,example2.jar -classpath ./example1.jar,./example2.jar com.example.WordCount input_table output_table; ``` ### Version Support - **MaxCompute MapReduce**：Native interface, fast execution speed - **MR2**：Extended version, supports more complex job scheduling logic ## Restrictions - JAR resources must be uploaded and published in advance before they can be referenced - For specific restrictions, see the MaxCompute official documentation ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_odps_mr", "script": { "path": "example_odps_mr", "runtime": { "command": "ODPS_MR" }, "content": "--MaxCompute MR\njar -resources mr_example.jar -classpath ./mr_example.jar com.example.WordCount;" } } ] } } ``` ## Reference - [MaxCompute MR Node](https://help.aliyun.com/zh/dataworks/user-guide/maxcompute-mr-node) FILE:references/nodetypes/maxcompute/ODPS_PERL.md # MaxCompute Perl（ODPS_PERL） ## Overview - Compute engine: `ODPS` - Content format: shell - Extension: `.mc.pl` - Code: 9 - Data source type: `odps` - Description: MaxCompute Perl script The MaxCompute Perl node is used to run Perl scripts in the MaxCompute scheduling environment, suitable for scenarios that require Perl for text processing or data transformation. ## Prerequisites - The workspace has been bound to a MaxCompute compute resource ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_odps_perl", "script": { "path": "example_odps_perl", "runtime": { "command": "odps_pl" }, "content": "#!/usr/bin/perl\nuse strict;\nprint \"MaxCompute Perl\\n\";" } } ] } } ``` FILE:references/nodetypes/maxcompute/ODPS_PYTHON.md # MaxCompute Python Resource（ODPS_PYTHON） ## Overview - Compute engine: `ODPS` - Content format: empty (no code) - Extension: `.json` - Code: 12 - Data source type: `odps` - Label type: RESOURCE - Description: MaxCompute Python resource The Python resource node is used to upload Python script files to a MaxCompute project for use by PyODPS nodes, Python UDFs, or Spark Python jobs. Resources must be published after upload before they can be used by other nodes. ## Prerequisites - The workspace has been bound to a MaxCompute compute resource ## Usage Notes - Resources must be published before they can be referenced by other nodes - Can serve as a dependency resource for Python UDFs - Referenced in PyODPS nodes via the `##@resource_reference{resource_name}` comment ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_odps_python", "script": { "path": "example_odps_python", "runtime": { "command": "ODPS_PYTHON" }, "content": "" } } ] } } ``` FILE:references/nodetypes/maxcompute/ODPS_SCRIPT.md # MaxCompute Script（ODPS_SCRIPT） ## Overview - Compute engine: `ODPS` - Content format: sql - Extension: `.ms` - Code: 24 - Data source type: `odps` - Description: MaxCompute Script (multi-statement script) The MaxCompute Script node is based on the MaxCompute 2.0 SQL engine and supports combining multiple SQL statements into a single script that is compiled and executed as a whole. By submitting once to generate a unified execution plan, it improves resource utilization efficiency. It is suitable for complex multi-step query scenarios. ## Prerequisites - The workspace has been bound to a MaxCompute compute resource - The required tables have been created and data added in MaxCompute ## Core Features ### Multi-statement Script Supports using the `@variable := SELECT...` syntax to define table variables. Variables can be used in subsequent JOIN, UNION, and other operations, enabling step-by-step processing of complex queries. ```sql -- MaxCompute Script example @user_base := SELECT user_id, user_name, city FROM ods_user_info WHERE dt = '${bizdate}'; @order_agg := SELECT user_id, SUM(amount) AS total_amount FROM ods_order WHERE dt = '${bizdate}' GROUP BY user_id; INSERT OVERWRITE TABLE dwd_user_order PARTITION (dt='${bizdate}') SELECT a.user_id, a.user_name, a.city, b.total_amount FROM @user_base a LEFT JOIN @order_agg b ON a.user_id = b.user_id; ``` ### Differences from Regular SQL Nodes | Dimension | MaxCompute Script | Regular ODPS_SQL | |------|------------------|--------------| | Execution mode | Compiled and executed as a whole | Executed statement by statement | | Number of jobs | Single job | Multiple jobs | | Applicable scenarios | Complex multi-step queries | Simple single-step operations | ## Restrictions - At most one SELECT statement with screen-displayed results is supported - At most one CREATE TABLE AS statement is supported (it must be the last statement) - If any statement fails, the entire script fails - Writing to and then reading from the same table is prohibited - Jobs are generated only after all input data is ready ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_odps_script", "script": { "path": "example_odps_script", "runtime": { "command": "ODPS_SQL_SCRIPT" }, "content": "--MaxCompute Script\nSELECT 1;" } } ] } } ``` ## Reference - [MaxCompute Script Node](https://help.aliyun.com/zh/dataworks/user-guide/maxcompute-script-node) FILE:references/nodetypes/maxcompute/ODPS_SHARK.md # MaxCompute Shark（ODPS_SHARK） ## Overview - Compute engine: `ODPS` - Content format: json - Extension: `.mc.shark.json` - Code: 223 - Data source type: `odps` - Description: MaxCompute Shark configuration The MaxCompute Shark node is used to configure and run Shark jobs. Shark is an early Hive-compatible query engine, and this node type is mainly retained for compatibility with existing tasks. ## Prerequisites - The workspace has been bound to a MaxCompute compute resource ## Content Structure ```json { "type": "JSON configuration object" } ``` ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_odps_shark", "script": { "path": "example_odps_shark", "runtime": { "command": "ODPS_SHARK" }, "content": "{}" } } ] } } ``` FILE:references/nodetypes/maxcompute/ODPS_SPARK.md # MaxCompute Spark（ODPS_SPARK） ## Overview - Compute engine: `ODPS` - Content format: json - Extension: `.mc.spark.json` - Code: 225 - Data source type: `odps` - Description: MaxCompute Spark job configuration The MaxCompute Spark node supports running Spark offline jobs in DataWorks through Cluster mode, supporting three development languages: Java, Scala, and Python. ## Prerequisites - The RAM account must have **Developer** or **Workspace Admin** role permissions - If selecting Spark 3.x version, a Serverless resource group must be purchased ## Core Features ### Configuration Items **Java/Scala jobs:** - Spark version (1.x / 2.x / 3.x) - Main JAR resource file - Main Class (fully qualified class name) - Configuration items, parameters, and associated JAR/File/Archives resources **Python jobs:** - Spark version (1.x / 2.x / 3.x) - Main Python resource file - Configuration items, parameters, and associated Python/File/Archives resources > No need to upload the spark-defaults.conf file; its configurations should be added individually to the node configuration items. ## Content Structure ```json { "mainClass": "entry class fully qualified name", "jars": ["JAR resource list"], "args": ["Runtime parameters"] } ``` ## Restrictions - Executes in Cluster mode; a custom program entry `main` must be specified - Python The default environment has limited dependency packages; a custom Python environment can be used - Spark 3.x version requires Serverless resource group support ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_odps_spark", "script": { "path": "example_odps_spark", "runtime": { "command": "ODPS_SPARK" }, "content": "{\"mainClass\":\"com.example.SparkJob\",\"jars\":[\"spark_job.jar\"]}" } } ] } } ``` ## Reference - [MaxCompute Spark Node](https://help.aliyun.com/zh/dataworks/user-guide/maxcompute-spark-node) FILE:references/nodetypes/maxcompute/ODPS_SQL.md # MaxCompute SQL（ODPS_SQL） ## Overview - Compute engine: `ODPS` - Content format: sql - Extension: `.sql` - Code: 10 - Data source type: `odps` - Description: MaxCompute SQL statements The MaxCompute SQL node is the most commonly used data processing node in DataWorks. It is used to execute standard SQL statements on MaxCompute (formerly ODPS). It is suitable for distributed processing scenarios involving massive data (TB-level) with low real-time requirements, supporting common operations such as DDL, DML, DQL, etc. ## Prerequisites - The workspace has been bound to a MaxCompute compute resource - The RAM account must have **Developer** or **Workspace Admin** role permissions ## Core Features ### SQL Syntax Supports MaxCompute SQL syntax, including DDL, DML, DQL statements, as well as built-in functions and user-defined functions (UDFs). Supports dynamic scheduling parameter input in the `${variable_name}` format. ```sql -- Create partitioned table CREATE TABLE IF NOT EXISTS dwd_user_info ( user_id BIGINT, user_name STRING, age INT ) PARTITIONED BY (dt STRING) LIFECYCLE 90; -- Partition write (using scheduling parameters) INSERT OVERWRITE TABLE dwd_user_info PARTITION (dt='${bizdate}') SELECT user_id, user_name, age FROM ods_user_info WHERE dt = '${bizdate}'; -- Aggregation query SELECT city, COUNT(*) AS cnt FROM dwd_user_info WHERE dt = '${bizdate}' GROUP BY city ORDER BY cnt DESC LIMIT 100; ``` ### Scheduling Parameters Supports defining dynamic parameters using the `${variable_name}` format. Values are assigned to variables in the scheduling configuration. Common system parameters include: - `${bizdate}`: Business date (yyyymmdd format) - `${yyyymmdd}`: Run date - `${yyyy-mm-dd}`: Run date (hyphenated format) ### Execution Notes - In Data Studio, all keyword statements are merged and executed upfront; in the scheduling environment, statements are executed in the actual written order - Only single-line comments `--` are supported ## Restrictions | Restriction | Requirement | |-------|------| | Code size | Up to 128 KB | | SQL command count | Up to 200 | | Query result rows | Up to 10,000 | | Query result size | Up to 10 MB | | Comment style | Only single-line comments `--` are supported | ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_odps_sql", "script": { "path": "example_odps_sql", "runtime": { "command": "ODPS_SQL" }, "content": "SELECT col1, COUNT(*) AS cnt FROM my_table WHERE dt='${bizdate}' GROUP BY col1;" } } ] } } ``` ### Complete Spec Example A complete node definition including scheduling, data source, and dependency configuration: ```json { "version": "1.1.0", "kind": "Node", "spec": { "nodes": [ { "name": "dwd_user_info", "recurrence": "Normal", "instanceMode": "T+1", "rerunMode": "Allowed", "script": { "path": "dwd_user_info", "language": "odps-sql", "runtime": { "command": "ODPS_SQL" }, "parameters": [ { "name": "bizdate", "scope": "NodeParameter", "type": "System", "value": "$yyyymmdd", "artifactType": "Variable" } ] }, "trigger": { "type": "Scheduler", "cron": "00 00 02 * * ?", "startTime": "1970-01-01 00:00:00", "endTime": "9999-01-01 00:00:00", "timezone": "Asia/Shanghai" }, "datasource": { "name": "odps_first", "type": "odps" }, "runtimeResource": { "resourceGroup": "S_res_group_XXX" } } ] } } ``` ## Reference - [MaxCompute SQL Node](https://help.aliyun.com/zh/dataworks/user-guide/maxcompute-sql-node) FILE:references/nodetypes/maxcompute/ODPS_TABLE.md # MaxCompute Table（ODPS_TABLE） ## Overview - Compute engine: `ODPS` - Content format: json - Extension: `.json` - Code: 16 - Data source type: `odps` - Label type: TABLE - Description: MaxCompute table definition The MaxCompute table node is used to define MaxCompute table structures in JSON format, including column definitions, partition columns, lifecycle, etc. This node enables version management and automated deployment of table structures. ## Prerequisites - The workspace has been bound to a MaxCompute compute resource ## Content Structure ```json { "name": "table name", "columns": [ {"name": "column name", "type": "data type"} ], "partitions": [ {"name": "partition column name", "type": "STRING"} ], "lifecycle": 90 } ``` ### Supported Data Types Common data types supported by MaxCompute include: BIGINT, STRING, DOUBLE, BOOLEAN, DATETIME, DECIMAL, INT, FLOAT, TIMESTAMP, BINARY, ARRAY, MAP, STRUCT, etc. ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_odps_table", "script": { "path": "example_odps_table", "runtime": { "command": "ODPS_TABLE" }, "content": "{\"name\":\"example_table\",\"columns\":[{\"name\":\"id\",\"type\":\"BIGINT\"},{\"name\":\"name\",\"type\":\"STRING\"}]}" } } ] } } ``` FILE:references/nodetypes/maxcompute/ODPS_XLIB.md # MaxCompute XLib（ODPS_XLIB） ## Overview - Compute engine: `ODPS` - Content format: python - Extension: `.mc.xlib.py` - Code: 8 - Data source type: `odps` - Description: MaxCompute XLib Python script The MaxCompute XLib node is used to run Python scripts that depend on extension libraries (XLib) on MaxCompute. Unlike PyODPS nodes, XLib nodes can use pre-installed scientific computing libraries such as NumPy. ## Prerequisites - The workspace has been bound to a MaxCompute compute resource ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_odps_xlib", "script": { "path": "example_odps_xlib", "runtime": { "command": "ODPS_XLIB" }, "content": "# MaxCompute XLib\nimport numpy as np\nprint(np.__version__)" } } ] } } ``` FILE:references/nodetypes/maxcompute/PYODPS.md # PyODPS 2（PYODPS） ## Overview - Compute engine: `ODPS` - Content format: python - Extension: `.py` - Code: 221 - Data source type: `odps` - Description: PyODPS 2 script (Python 2, operates on MaxCompute) The PyODPS 2 node is based on the MaxCompute Python SDK (PyODPS) and allows writing code in Python 2 to operate on MaxCompute. It supports executing SQL, DataFrame processing, and resource management. > **Recommendation**: It is recommended that new projects use [PYODPS3](./PYODPS3.md) (Python 3 version). ## Prerequisites - The workspace has been bound to a MaxCompute compute resource ## Core Features ### Built-in Global Variables The following global variables are preset in the node and do not need to be defined manually: - **`odps` or `o`**: ODPS entry object for directly calling MaxCompute APIs - **`args`**: Dictionary-format scheduling parameter container, used to retrieve parameter values such as `${yyyymmdd}` ```python # PyODPS 2 example print(o.exist_table('my_table')) # Get scheduling parameters bizdate = args['bizdate'] print('bizdate: ' + bizdate) # Execute SQL with o.execute_sql('SELECT COUNT(*) FROM my_table').open_reader() as reader: print(reader[0][0]) ``` ## Restrictions | Restriction | Description | |-------|------| | Python version | Underlying Python 2.7 | | Local data volume | Dedicated resource groups: up to 50MB; Serverless resource groups: 16 CU or less recommended | | Log size | Maximum output log size is 4MB | | Instance Tunnel | Disabled by default; manually set `options.tunnel.use_instance_tunnel = True` | | Concurrent execution | Concurrent execution of multiple Python tasks is not supported | | Log output | Only `print` is supported; `logger.info` is not supported | ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_pyodps", "script": { "path": "example_pyodps", "runtime": { "command": "PY_ODPS" }, "content": "from odps import ODPS\nimport sys\nprint('PyODPS 2 node')\nprint(sys.version)" } } ] } } ``` ## Reference - [PyODPS 2 Node](https://help.aliyun.com/zh/dataworks/user-guide/pyodps-2-node) FILE:references/nodetypes/maxcompute/PYODPS3.md # PyODPS 3（PYODPS3） ## Overview - Compute engine: `ODPS` - Content format: python - Extension: `.py` - Code: 1221 - Data source type: `odps` - Description: PyODPS 3 script (Python 3, operates on MaxCompute) The PyODPS 3 node is based on the MaxCompute Python SDK (PyODPS) and allows writing code in Python 3 to operate on MaxCompute. It supports executing SQL, DataFrame processing, and resource management. It is recommended that new projects use this node type instead of [PYODPS](./PYODPS.md). ## Prerequisites - The workspace has been bound to a MaxCompute compute resource ## Core Features ### Built-in Global Variables The following global variables are preset in the node and do not need to be defined manually: - **`odps` or `o`**: ODPS entry object for directly calling MaxCompute APIs - **`args`**: Dictionary-format scheduling parameter container ```python # PyODPS 3 example print(o.exist_table('my_table')) # Get scheduling parameters bizdate = args['bizdate'] print(f'bizdate: {bizdate}') # Execute SQL with o.execute_sql('SELECT COUNT(*) FROM my_table').open_reader() as reader: print(reader[0][0]) ``` ## Restrictions | Restriction | Description | |-------|------| | Python version | MaxCompute uses Python 3.7; using 3.7 locally is recommended to avoid bytecode incompatibility | | Local data volume | Dedicated resource groups: up to 50MB; Serverless resource groups: 16 CU or less recommended | | Log size | Maximum output log size is 4MB | | Concurrent execution | Concurrent execution of multiple Python tasks is not supported | | Log output | Only `print` is supported; `logger.info` is not supported | | Third-party packages | Third-party packages with binary code are not supported (except Numpy and Pandas) | ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_pyodps3", "script": { "path": "example_pyodps3", "runtime": { "command": "PYODPS3" }, "content": "from odps import ODPS\nimport sys\nprint('PyODPS 3 node')\nprint(sys.version)" } } ] } } ``` ## Reference - [PyODPS 3 Node](https://help.aliyun.com/zh/dataworks/user-guide/pyodps-3-node) FILE:references/nodetypes/maxcompute/SQL_COMPONENT.md # SQL Component (New)（SQL_COMPONENT） ## Overview - Compute engine: `ODPS` - Content format: sql - Extension: `.sql` - Code: 3010 - Data source type: `odps` - Description: SQL component (new version), supports parameterized SQL The SQL component (new version) node provides SQL code templates with multiple input and output parameters. It generates result tables by performing filtering, joining, and aggregation operations on data source tables, helping developers quickly build reusable data processing nodes. ## Prerequisites - Only supported on DataWorks **Standard Edition or above** - Development permissions for the workspace are required - The workspace has been bound to a MaxCompute compute resource ## Core Features ### Parameterized SQL The component supports defining input parameters using `${parameter_name}`. When referencing the component, the system automatically identifies parameters and requires value assignment. ```sql -- SQL component example: parameterized query template SELECT ${columns} FROM ${source_table} WHERE dt = '${bizdate}' AND ${filter_condition}; ``` ### Version Management Supports upgrading component versions through the "Update Code Version" feature. Consumers can choose to use the new version. ## Restrictions - When accessing public network or VPC data sources, a scheduling resource group that has passed the connectivity test must be used ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_sql_component", "script": { "path": "example_sql_component", "runtime": { "command": "SQL_COMPONENT" }, "content": "SELECT ${param1} FROM dual;" } } ] } } ``` ## Reference - [SQL Component Node](https://help.aliyun.com/zh/dataworks/user-guide/sql-component-node) FILE:references/nodetypes/maxcompute/YSF_DESEN.md # MaxCompute Data Masking（YSF_DESEN） ## Overview - Compute engine: `ODPS` - Content format: sql - Extension: `.mc.data.masking.sql` - Code: 82 - Data source type: `odps` - Description: MaxCompute data masking SQL The data masking node is used to mask sensitive data in MaxCompute. It defines masking rules through SQL statements and automatically performs masking operations such as obfuscation and replacement on specified fields during data querying or processing. ## Prerequisites - The workspace has been bound to a MaxCompute compute resource - Data masking rules have been configured ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_ysf_desen", "script": { "path": "example_ysf_desen", "runtime": { "command": "YSF_DESEN" }, "content": "SELECT * FROM my_table;" } } ] } } ``` FILE:references/nodetypes/serverless_spark/SERVERLESS_KYUUBI.md # Serverless Kyuubi SQL (SERVERLESS_KYUUBI) ## Overview - Compute engine: `EMR` - Content format: sql - Extension: `.sql` - Data source type: `emr` - Code: 2103 - Description: Serverless Kyuubi SQL node, based on the Apache Kyuubi multi-tenant SQL gateway The Serverless Kyuubi SQL node is used to submit Spark SQL queries through the Apache Kyuubi multi-tenant SQL gateway. Kyuubi is a distributed multi-tenant gateway that provides Serverless SQL on Spark capabilities, supporting multiple users sharing Spark engine resources with better resource isolation and session management features. ## Prerequisites - An EMR Serverless Spark compute resource has been bound, with the Kyuubi service enabled - Ensure the resource group has network connectivity with the compute resource - Only Serverless resource groups are supported for running this task type - RAM users must have **Developer** or **Workspace Admin** role permissions ## Core Features ### SQL Syntax Kyuubi SQL supports the full Spark SQL syntax, submitted for execution through the Kyuubi gateway: ```sql -- Data query SELECT dt, COUNT(DISTINCT user_id) AS uv, COUNT(*) AS pv FROM user_log WHERE dt = '${bizdate}' GROUP BY dt; -- Create table CREATE TABLE IF NOT EXISTS dwd_order_detail ( order_id BIGINT COMMENT 'Order ID', user_id BIGINT COMMENT 'User ID', amount DECIMAL(10, 2) COMMENT 'Order amount', order_time TIMESTAMP COMMENT 'Order time' ) PARTITIONED BY (dt STRING) STORED AS PARQUET; -- Data write INSERT OVERWRITE TABLE dwd_order_detail PARTITION (dt = '${bizdate}') SELECT order_id, user_id, amount, order_time FROM ods_order WHERE dt = '${bizdate}'; ``` ### Scheduling Parameters Supports defining dynamic parameters using `${variable_name}` syntax, with values assigned in the scheduling configuration. ### Differences from Serverless Spark SQL | Feature | Kyuubi SQL | Spark SQL | |------|-----------|-----------| | Execution engine | Via Kyuubi gateway | Direct Spark SQL | | Multi-tenant | Natively supported | Requires additional configuration | | Session management | Kyuubi session pool | Independent session | | Resource sharing | Shared Spark engine | Independent engine | | Applicable scenarios | Multi-user interactive queries | Batch SQL tasks | ### Advantages - **Multi-tenant isolation**: Supports multiple simultaneous users with mutual isolation - **Session reuse**: Reuses Spark engines via session pool mechanism, reducing startup time - **Standard interfaces**: Compatible with JDBC/ODBC standard protocols - **Resource elasticity**: Based on Serverless architecture, resources are allocated on demand ## Restrictions - Only Serverless resource groups are supported for execution - The EMR Serverless Spark cluster must have the Kyuubi service enabled - SQL syntax is limited by the Spark SQL version ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_serverless_kyuubi", "script": { "path": "example_serverless_kyuubi", "runtime": { "command": "SERVERLESS_KYUUBI" }, "content": "SELECT 1;" } } ] } } ``` ## Related Documentation - [Serverless Kyuubi SQL Node](https://help.aliyun.com/zh/dataworks/user-guide/serverless-kyuubi-node) FILE:references/nodetypes/serverless_spark/SERVERLESS_PYSPARK.md # Serverless PySpark (SERVERLESS_PYSPARK) ## Overview - Compute engine: `EMR` - Content format: python - Extension: `.py` - Data source type: `emr` - Code: 2105 - Description: Serverless PySpark node, Python-based distributed computing on EMR Serverless Spark DataWorks provides the Serverless PySpark node, allowing users to directly develop and run distributed PySpark tasks based on EMR Serverless Spark without the need to manage cluster infrastructure. This node uses a dual-panel collaborative editing mode: The upper panel is for writing Python business logic, and the lower panel is for writing the `spark-submit` submission command. ## Prerequisites - An EMR Serverless Spark compute resource has been bound, Ensure the resource group has network connectivity with the compute resource - Only Serverless resource groups are supported for running this task type - RAM users must have **Developer** or **Workspace Admin** role permissions ## Core Features ### Dual-panel Editing Mode **Upper Panel -- Python Business Code**: Declare referenced external Python files via `##@resource_reference{"resource_name"}`. ```python ##@resource_reference{"utils.py"} from pyspark.sql import SparkSession from utils import estimate_pi_in_task import sys def main(): spark = SparkSession.builder.appName("EstimatePi").getOrCreate() sc = spark.sparkContext total_samples = int(sys.argv[1]) num_partitions = ${test1} # Parallel computation counts = sc.parallelize(range(num_partitions), num_partitions) \ .map(lambda i: estimate_pi_in_task(total_samples // num_partitions)) \ .reduce(lambda a, b: a + b) pi = 4.0 * counts / total_samples print(f"Pi is approximately {pi}") spark.stop() if __name__ == "__main__": main() ``` **Lower Panel -- spark-submit Command**: ```bash spark-submit \ --py-files utils.py \ serverless_pyspark_test1.py 10000 ``` ### Scheduling Parameters - Define dynamic parameters using `${variable_name}` format (e.g., `${test1}` in the example) - Receive spark-submit command-line parameters via `sys.argv` - Assign values to variables in the scheduling configuration ### Resource Reference External Python files must first be uploaded via the resource management module, then: 1. Declare references in code using `##@resource_reference{"resource_name"}` 2. Explicitly declare dependency files in the spark-submit command via `--py-files` ## Restrictions - The main Python script filename must match the node name, with `.py` suffix - External `.py` files must be explicitly declared via `--py-files` - Only submitting the entire Python file is supported; running partial code is not supported - Only Serverless resource groups are supported for execution ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_serverless_pyspark", "script": { "path": "example_serverless_pyspark", "runtime": { "command": "SERVERLESS_PYSPARK" }, "content": "print('hello')" } } ] } } ``` ## Related Documentation - [Serverless PySpark Node](https://help.aliyun.com/zh/dataworks/user-guide/serverless-pyspark-node) FILE:references/nodetypes/serverless_spark/SERVERLESS_SPARK_BATCH.md # Serverless Spark Batch (SERVERLESS_SPARK_BATCH) ## Overview - Compute engine: `EMR` - Content format: shell - Extension: `.sh` - Data source type: `emr` - Code: 2100 - Description: Serverless Spark Batch node, batch processing jobs based on EMR Serverless Spark The Serverless Spark Batch node is used in DataWorks to submit and schedule EMR Serverless Spark batch processing jobs. It uses Shell scripts to write `spark-submit` commands, submitting Spark JAR or PySpark jobs to the Serverless Spark cluster for execution, without the need to manage underlying cluster infrastructure. ## Prerequisites - An EMR Serverless Spark compute resource has been bound, Ensure the resource group has network connectivity with the compute resource - Only Serverless resource groups are supported for running this task type - The Spark job JAR package or Python file has been uploaded to OSS or DataWorks resource management - RAM users must have **Developer** or **Workspace Admin** role permissions ## Core Features ### Script Format The node content is a Shell script, typically containing the `spark-submit` command: ```bash #!/bin/bash # Spark Batch job submission spark-submit \ --class com.example.MySparkJob \ --master yarn \ --deploy-mode cluster \ --driver-memory 2g \ --executor-memory 4g \ --executor-cores 2 \ --num-executors 10 \ --conf spark.sql.shuffle.partitions=200 \ my_spark_job.jar \ --input oss://my-bucket/input/ \ --output oss://my-bucket/output/ \ --date ${bizdate} ``` ### Scheduling Parameters Supports defining dynamic parameters using `${variable_name}` syntax: ```bash spark-submit \ --class com.example.ETLJob \ my_etl.jar \ --date ${bizdate} \ --partition ${partition_value} ``` ### Applicable Scenarios - **Large-scale ETL**: Batch data processing based on the Spark engine - **Machine learning**: Submit Spark MLlib training jobs - **Data analysis**: Complex distributed computing tasks - **Custom jobs**: Complex business logic that cannot be fulfilled by SQL ### Resource Configuration Control resource allocation through `spark-submit` parameters: | Parameter | Description | |------|------| | `--driver-memory` | Driver process memory | | `--executor-memory` | Memory per Executor | | `--executor-cores` | CPU cores per Executor | | `--num-executors` | Number of Executors | ## Restrictions - Only Serverless resource groups are supported for execution - Job JAR packages or dependency files must be uploaded in advance - The `spark-submit` parameters in the Shell script must comply with Serverless Spark specifications ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_serverless_spark_batch", "script": { "path": "example_serverless_spark_batch", "runtime": { "command": "SERVERLESS_SPARK_BATCH" }, "content": "#!/bin/bash\necho 'hello'" } } ] } } ``` ## Related Documentation - [Serverless Spark Batch Node](https://help.aliyun.com/zh/dataworks/user-guide/serverless-spark-batch-node) FILE:references/nodetypes/serverless_spark/SERVERLESS_SPARK_SQL.md # Serverless Spark SQL (SERVERLESS_SPARK_SQL) ## Overview - Compute engine: `EMR` - Content format: sql - Extension: `.sql` - Data source type: `emr` - Code: 2101 - Description: Serverless Spark SQL node, distributed SQL queries based on EMR Serverless Spark By creating a Serverless Spark SQL node, you can process structured data using the distributed SQL query engine based on EMR Serverless Spark compute resources, improving job execution efficiency. This node does not require managing cluster infrastructure and uses the Spark SQL engine to execute queries on demand. ## Prerequisites - An EMR Serverless Spark compute resource has been bound, Ensure the resource group has network connectivity with the compute resource - Only Serverless resource groups are supported for running this task type - RAM users must have **Developer** or **Workspace Admin** role permissions (can be ignored for primary accounts) ## Core Features ### SQL Syntax Supports the full `catalog.database.tablename` syntax. If the `catalog` parameter is omitted, the system uses the cluster's default Catalog; if `catalog.database` is omitted, the default database of the default Catalog is used. ```sql -- Basic query SELECT * FROM ; -- Create table (using scheduling parameters) CREATE TABLE IF NOT EXISTS userinfo_new_${var}( ip STRING COMMENT 'IP address', uid STRING COMMENT 'User ID' ) PARTITIONED BY (dt STRING); -- Data query and aggregation SELECT dt, COUNT(DISTINCT uid) AS uv FROM user_log WHERE dt = '${bizdate}' GROUP BY dt; ``` ### Scheduling Parameters Define dynamic variables using `${variable_name}` syntax, with values assigned in the scheduling configuration. Supports scheduling parameter expressions such as `${yyyymmdd}` for dynamic parameter passing. ### Advanced Parameters | Parameter | Description | Default value | |------|------|--------| | `FLOW_SKIP_SQL_ANALYZE` | `true` executes multiple SQL statements at once; `false` executes them one by one | `false` | | `DATAWORKS_SESSION_DISABLE` | `true` submits to the queue for execution; `false` executes via SQL Compute | `false` | | `SERVERLESS_RELEASE_VERSION` | Specifies the Spark engine version | Cluster default version | | `SERVERLESS_QUEUE_NAME` | Specifies the resource queue for task submission | Default queue | | `SERVERLESS_SQL_COMPUTE` | Specifies the SQL session | Cluster default session | ## Restrictions - SQL statements must not exceed **130KB** - Only Serverless resource groups are supported for execution ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_serverless_spark_sql", "script": { "path": "example_serverless_spark_sql", "runtime": { "command": "SERVERLESS_SPARK_SQL" }, "content": "SELECT 1;" } } ] } } ``` ## Related Documentation - [Serverless Spark SQL Node](https://help.aliyun.com/zh/dataworks/user-guide/serverless-spark-sql-node) FILE:references/nodetypes/serverless_spark/SERVERLESS_SPARK_STREAMING.md # Serverless Spark Streaming (SERVERLESS_SPARK_STREAMING) ## Overview - Compute engine: `EMR` - Content format: shell - Extension: `.sh` - Data source type: `emr` - Code: 2102 - Description: Serverless Spark Streaming node, stream processing jobs based on EMR Serverless Spark The Serverless Spark Streaming node is used in DataWorks to submit and manage real-time stream processing jobs based on Spark Structured Streaming. It uses Shell scripts to write `spark-submit` commands, submitting stream processing jobs to the Serverless Spark cluster, enabling continuous consumption and processing of real-time data. ## Prerequisites - An EMR Serverless Spark compute resource has been bound, Ensure the resource group has network connectivity with the compute resource - Only Serverless resource groups are supported for running this task type - The Spark Streaming job JAR package has been uploaded to OSS or DataWorks resource management - RAM users must have **Developer** or **Workspace Admin** role permissions ## Core Features ### Script Format The node content is a Shell script, typically containing the `spark-submit` command for streaming jobs: ```bash #!/bin/bash # Spark Streaming job submission spark-submit \ --class com.example.MyStreamingJob \ --master yarn \ --deploy-mode cluster \ --driver-memory 2g \ --executor-memory 4g \ --executor-cores 2 \ --num-executors 5 \ --conf spark.streaming.kafka.maxRatePerPartition=1000 \ --conf spark.sql.streaming.checkpointLocation=oss://my-bucket/checkpoint/ \ my_streaming_job.jar \ --bootstrap-servers kafka-broker:9092 \ --topic user_events \ --output-table hologres_sink_table ``` ### Applicable Scenarios - **Real-time ETL**: Consume data from Kafka/DataHub and process/write to target tables in real time - **Real-time monitoring**: Continuously monitor data streams and trigger alerts - **Real-time aggregation**: Window aggregation based on Spark Structured Streaming - **Data sync**: Sync data from one storage to another in real time ### Differences from Flink Streaming Nodes | Feature | Spark Streaming | Flink Streaming | |------|----------------|-----------------| | Processing model | Micro-batch | True stream processing | | Latency | Second-level | Millisecond-level | | Programming model | Spark API | Flink SQL / DataStream | | Applicable scenarios | Near-real-time ETL, micro-batch processing | Low-latency real-time computation | ## Restrictions - Streaming tasks run continuously after startup until manually stopped or abnormally terminated - Only Serverless resource groups are supported for execution - Checkpoints must be properly configured to ensure fault tolerance - Job JAR packages must be uploaded in advance ## Minimum Spec ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "example_serverless_spark_streaming", "script": { "path": "example_serverless_spark_streaming", "runtime": { "command": "SERVERLESS_SPARK_STREAMING" }, "content": "#!/bin/bash\necho 'hello'" } } ] } } ``` FILE:references/ram-policies.md # DataWorks Data Development RAM Permission List This document lists all RAM permissions required to use the DataWorks data development SKILL. ## Required Permissions ### Project Management Permissions | Permission | Description | API | |-----|------|-----| | `dataworks:GetProject` | Get project information | GetProject | ### Node Management Permissions | Permission | Description | API | |-----|------|-----| | `dataworks:ListNodes` | List nodes | ListNodes | | `dataworks:GetNode` | Get node details | GetNode | | `dataworks:CreateNode` | Create node | CreateNode | | `dataworks:UpdateNode` | Update node | UpdateNode | | `dataworks:MoveNode` | Move a node to a specified path | MoveNode | | `dataworks:RenameNode` | Rename a node | RenameNode | | `dataworks:ListNodeDependencies` | List a node's dependency nodes | ListNodeDependencies | ### Workflow Management Permissions | Permission | Description | API | |-----|------|-----| | `dataworks:ListWorkflowDefinitions` | List workflows | ListWorkflowDefinitions | | `dataworks:GetWorkflowDefinition` | Get workflow details | GetWorkflowDefinition | | `dataworks:CreateWorkflowDefinition` | Create workflow | CreateWorkflowDefinition | | `dataworks:UpdateWorkflowDefinition` | Update workflow | UpdateWorkflowDefinition | | `dataworks:ImportWorkflowDefinition` | Import a workflow definition | ImportWorkflowDefinition | | `dataworks:MoveWorkflowDefinition` | Move a workflow to a target path | MoveWorkflowDefinition | | `dataworks:RenameWorkflowDefinition` | Rename a workflow | RenameWorkflowDefinition | ### Deployment Management Permissions | Permission | Description | API | |-----|------|-----| | `dataworks:CreatePipelineRun` | Create deployment process | CreatePipelineRun | | `dataworks:GetPipelineRun` | Get deployment status | GetPipelineRun | | `dataworks:ExecPipelineRunStage` | Advance deployment stage | ExecPipelineRunStage | | `dataworks:ListPipelineRuns` | Query deployment history | ListPipelineRuns | | `dataworks:ListPipelineRunItems` | Query deployment items | ListPipelineRunItems | | `dataworks:AbolishPipelineRun` | Cancel deployment | AbolishPipelineRun | ### Data Source Management Permissions | Permission | Description | API | |-----|------|-----| | `dataworks:ListDataSources` | List data sources | ListDataSources | ### Resource Group Management Permissions | Permission | Description | API | |-----|------|-----| | `dataworks:ListResourceGroups` | List resource groups | ListResourceGroups | ### Resource Management Permissions | Permission | Description | API | |-----|------|-----| | `dataworks:CreateResource` | Create a file resource | CreateResource | | `dataworks:UpdateResource` | Update file resource information | UpdateResource | | `dataworks:MoveResource` | Move a file resource to a specified directory | MoveResource | | `dataworks:RenameResource` | Rename a file resource | RenameResource | | `dataworks:GetResource` | Get file resource details | GetResource | | `dataworks:ListResources` | List file resources | ListResources | ### Function Management Permissions | Permission | Description | API | |-----|------|-----| | `dataworks:CreateFunction` | Create a UDF function | CreateFunction | | `dataworks:UpdateFunction` | Update UDF function information | UpdateFunction | | `dataworks:MoveFunction` | Move a function to a target path | MoveFunction | | `dataworks:RenameFunction` | Rename a function | RenameFunction | | `dataworks:GetFunction` | Get function details | GetFunction | | `dataworks:ListFunctions` | List functions | ListFunctions | ### Component Management Permissions | Permission | Description | API | |-----|------|-----| | `dataworks:CreateComponent` | Create a component | CreateComponent | | `dataworks:GetComponent` | Get component details | GetComponent | | `dataworks:UpdateComponent` | Update a component | UpdateComponent | | `dataworks:ListComponents` | List components | ListComponents | ## Recommended Policies ### Minimum Permission Policy (Read-Only) ```json { "Version": "1", "Statement": [ { "Effect": "Allow", "Action": [ "dataworks:GetProject", "dataworks:ListNodes", "dataworks:GetNode", "dataworks:ListNodeDependencies", "dataworks:ListWorkflowDefinitions", "dataworks:GetWorkflowDefinition", "dataworks:GetPipelineRun", "dataworks:ListPipelineRuns", "dataworks:ListPipelineRunItems", "dataworks:ListDataSources", "dataworks:ListResourceGroups", "dataworks:GetResource", "dataworks:ListResources", "dataworks:GetFunction", "dataworks:ListFunctions", "dataworks:GetComponent", "dataworks:ListComponents" ], "Resource": "*" } ] } ``` ### Full Development Permission Policy ```json { "Version": "1", "Statement": [ { "Effect": "Allow", "Action": [ "dataworks:GetProject", "dataworks:ListNodes", "dataworks:GetNode", "dataworks:CreateNode", "dataworks:UpdateNode", "dataworks:MoveNode", "dataworks:RenameNode", "dataworks:ListNodeDependencies", "dataworks:ListWorkflowDefinitions", "dataworks:GetWorkflowDefinition", "dataworks:CreateWorkflowDefinition", "dataworks:UpdateWorkflowDefinition", "dataworks:ImportWorkflowDefinition", "dataworks:MoveWorkflowDefinition", "dataworks:RenameWorkflowDefinition", "dataworks:CreatePipelineRun", "dataworks:GetPipelineRun", "dataworks:ExecPipelineRunStage", "dataworks:ListPipelineRuns", "dataworks:ListPipelineRunItems", "dataworks:AbolishPipelineRun", "dataworks:ListDataSources", "dataworks:ListResourceGroups", "dataworks:CreateResource", "dataworks:UpdateResource", "dataworks:MoveResource", "dataworks:RenameResource", "dataworks:GetResource", "dataworks:ListResources", "dataworks:CreateFunction", "dataworks:UpdateFunction", "dataworks:MoveFunction", "dataworks:RenameFunction", "dataworks:GetFunction", "dataworks:ListFunctions", "dataworks:CreateComponent", "dataworks:GetComponent", "dataworks:UpdateComponent", "dataworks:ListComponents" ], "Resource": "*" } ] } ``` ## Restrict Permissions by Project To restrict permissions to a specific project, change `Resource` to the project ARN: ```json { "Version": "1", "Statement": [ { "Effect": "Allow", "Action": [ "dataworks:CreateNode" ], "Resource": [ "acs:dataworks:cn-hangzhou:123456789012:project/my_project_name" ] } ] } ``` ## Common Permission Errors | Error Code | Description | Solution | |-------|------|---------| | `Forbidden.RAM` | Insufficient permissions | Add the corresponding API permission | | `NoPermission` | No operation permission | Check if the RAM policy is in effect | | `InvalidAccessKeyId.NotFound` | Invalid AccessKey | Check AccessKey configuration | | `SignatureDoesNotMatch` | Signature mismatch | Check AccessKeySecret | ## References - [DataWorks RAM Permission Guide](https://help.aliyun.com/zh/dataworks/user-guide/dataworks-ram-permissions) - [RAM Policy Management](https://ram.console.aliyun.com/policies) FILE:references/references.md FILE:references/related-commands.md # DataWorks Related CLI Commands This document lists all CLI commands involved in the DataWorks data development SKILL. ## Node Operations | Product | CLI Command | Description | |---------|-------------|-------------| | dataworks-public | `aliyun dataworks-public CreateNode` | Create node | | dataworks-public | `aliyun dataworks-public UpdateNode` | Update node | | dataworks-public | `aliyun dataworks-public GetNode` | Get node details | | dataworks-public | `aliyun dataworks-public ListNodes` | List nodes | ## Workflow Operations | Product | CLI Command | Description | |---------|-------------|-------------| | dataworks-public | `aliyun dataworks-public CreateWorkflowDefinition` | Create workflow | | dataworks-public | `aliyun dataworks-public UpdateWorkflowDefinition` | Update workflow | | dataworks-public | `aliyun dataworks-public GetWorkflowDefinition` | Get workflow details | | dataworks-public | `aliyun dataworks-public ListWorkflowDefinitions` | List workflows | ## Deployment Operations | Product | CLI Command | Description | |---------|-------------|-------------| | dataworks-public | `aliyun dataworks-public CreatePipelineRun` | Create deployment process | | dataworks-public | `aliyun dataworks-public GetPipelineRun` | Get deployment status | | dataworks-public | `aliyun dataworks-public ExecPipelineRunStage` | Advance deployment stage | | dataworks-public | `aliyun dataworks-public ListPipelineRuns` | Query deployment history | | dataworks-public | `aliyun dataworks-public ListPipelineRunItems` | Query deployment items | | dataworks-public | `aliyun dataworks-public AbolishPipelineRun` | Cancel deployment | ## Project and Resource Operations | Product | CLI Command | Description | |---------|-------------|-------------| | dataworks-public | `aliyun dataworks-public GetProject` | Get project information | | dataworks-public | `aliyun dataworks-public ListDataSources` | List data sources | | dataworks-public | `aliyun dataworks-public ListResourceGroups` | List resource groups | ## Resource and Function Operations | Product | CLI Command | Description | |---------|-------------|-------------| | dataworks-public | `aliyun dataworks-public CreateResource` | Create resource | | dataworks-public | `aliyun dataworks-public ListResources` | List resources | | dataworks-public | `aliyun dataworks-public CreateFunction` | Create function | | dataworks-public | `aliyun dataworks-public ListFunctions` | List functions | ## Command Usage Examples ### Create Node ```bash aliyun dataworks-public CreateNode \ --ProjectId {{project_id}} \ --Scene DATAWORKS_PROJECT \ --Spec "$(cat /tmp/spec.json)" \ --user-agent AlibabaCloud-Agent-Skills ``` ### Create Node Within a Workflow ```bash aliyun dataworks-public CreateNode \ --ProjectId {{project_id}} \ --Scene DATAWORKS_PROJECT \ --ContainerId {{workflow_id}} \ --Spec "$(cat /tmp/spec.json)" \ --user-agent AlibabaCloud-Agent-Skills ``` ### Create Workflow ```bash aliyun dataworks-public CreateWorkflowDefinition \ --ProjectId {{project_id}} \ --Spec "$(cat /tmp/wf.json)" \ --user-agent AlibabaCloud-Agent-Skills ``` ### Deploy (Online) ```bash aliyun dataworks-public CreatePipelineRun \ --ProjectId {{project_id}} \ --Type Online \ --ObjectIds '["{{object_id}}"]' \ --user-agent AlibabaCloud-Agent-Skills ``` ### Query Deployment Status ```bash aliyun dataworks-public GetPipelineRun \ --ProjectId {{project_id}} \ --Id {{pipeline_run_id}} \ --user-agent AlibabaCloud-Agent-Skills ``` ### Advance Deployment Stage ```bash aliyun dataworks-public ExecPipelineRunStage \ --ProjectId {{project_id}} \ --Id {{pipeline_run_id}} \ --Code {{stage_code}} \ --user-agent AlibabaCloud-Agent-Skills ``` ## Command Help View command details: ```bash aliyun dataworks-public CreateNode --help aliyun dataworks-public ListNodes --help aliyun dataworks-public CreateWorkflowDefinition --help ``` ## Important Notes 1. **All commands must include `--user-agent AlibabaCloud-Agent-Skills`** 2. API version is `2024-05-18`, using the `dataworks-public` product 3. Parameter names are case-sensitive (e.g., `--ProjectId` not `--projectId`) FILE:references/scheduling-guide.md # Scheduling Configuration Guide Scheduling defines the frequency at which nodes automatically execute in the production environment. The DataWorks scheduling system automatically generates cycle instances based on the configured scheduling cycle, and triggers execution based on inter-node dependencies and the scheduled time of each instance. ## Core Concepts ### Cycle Instances The scheduling system generates a runtime entity, called a cycle instance, for each business date based on the scheduling configuration of a cycle task (e.g., run daily at midnight). The task's execution, status, and logs are all associated with this instance. ### Cross-Cycle Dependencies DataWorks supports dependencies between nodes with different scheduling cycles. For example, a daily downstream node can depend on an hourly upstream node. The essence of inter-node dependencies is the dependency between the cycle instances they generate. ### Dry Run For non-daily scheduled tasks (e.g., weekly, monthly, yearly), on dates that are not designated run dates, the scheduling system generates a "dry run" instance. This instance immediately changes to "Success" status once its scheduled time arrives, but **does not execute the node's code logic**. Dry runs primarily serve to bridge dependencies, ensuring that downstream daily nodes can trigger normally. - Instance run status is Success, execution duration is 0 seconds, no execution logs - Does not consume scheduling or compute resources - Does not block downstream node execution ## Scheduling Execution Conditions A cycle instance must **simultaneously satisfy** both of the following conditions to execute: 1. All upstream instances it depends on have successfully executed (including dry-run-success instances) 2. The instance's own scheduled time has been reached Therefore, the configured scheduling time is only the **expected scheduled time**; the actual execution time of a node is affected by upstream completion time, available resources, actual execution conditions, and other factors. ## Scheduling Differences: Workflows vs. Standalone Nodes - **Workflow nodes**: Scheduling time is configured uniformly on the workflow; individual nodes within the workflow cannot have their own scheduling times - **Standalone nodes**: Scheduling time is independently configured on the node itself If a node's scheduling cycle differs from other nodes in the workflow, it should be created as a standalone node or placed in a separate workflow. ## Workflow Scheduling Scenarios ### Scenario 1: Unified Start Time Workflow A->B->C, the entire business flow must start after 3:00 AM. Simply set the start node A's scheduled time to `03:00`. Even if downstream nodes B and C have a default scheduled time of `00:00`, they must wait until A successfully executes at 3:00 AM before starting sequentially. ### Scenario 2: Different Start Times for Each Node Node A is scheduled at 3:00 AM, Node B is required after 5:00 AM, Node C is required after 6:00 AM. Set the scheduled times for A, B, and C to `03:00`, `05:00`, and `06:00` respectively. ### Scenario 3: Some Nodes Have Specific Start Times Node A is scheduled at 3:00 AM, Node B is required after 5:00 AM, Node C has no specific requirement. Set A to `03:00` and B to `05:00`. Node C will wait for B to successfully execute after 5:00 AM, then start. ## Scheduling Types and Cron Expressions DataWorks supports six scheduling cycles: minute, hourly, daily, weekly, monthly, and yearly. ### Daily Scheduling Runs once at a specified time each day. ``` cron: 00 00 03 * * ? # Daily at 03:00 cron: 00 30 08 * * ? # Daily at 08:30 ``` ### Hourly Scheduling Runs at intervals within a specified time range. The time range follows the **closed-closed** principle. ``` cron: 00 00 */6 * * ? # Every 6 hours (00:00, 06:00, 12:00, 18:00) cron: 00 00 * * * ? # Every hour ``` ### Minute Scheduling Minimum interval is 1 minute. ``` cron: 00 */30 * * * ? # Every 30 minutes cron: 00 */5 * * * ? # Every 5 minutes ``` ### Weekly Scheduling A **dry run** occurs on non-designated scheduling dates. ``` cron: 00 00 01 ? * MON # Every Monday at 01:00 cron: 00 00 03 ? * MON,FRI # Every Monday and Friday at 03:00 ``` ### Monthly Scheduling A **dry run** occurs on non-designated dates. Supports the last day of the month. ``` cron: 00 00 02 1 * ? # 1st of every month at 02:00 cron: 00 00 02 L * ? # Last day of every month at 02:00 ``` ### Yearly Scheduling A **dry run** occurs on non-designated dates. ``` cron: 00 00 02 1 1,4,7,10 ? # 1st of January/April/July/October at 02:00 ``` ## Impact of Updating Scheduling Time After modifying a node's scheduling time and re-deploying, the impact depends on the instance generation method: - **T+1 next-day generation**: The scheduled times of already-generated instances for the most recent two days (T and T-1) will be updated to the new configuration. Future instances are generated with the new time - **Immediate instance generation**: New instances are generated immediately based on the new configuration; historical instances remain unchanged ## Business Date In DataWorks, **business date = scheduled run date - 1**. For example: - To backfill a weekly task that runs on Monday, the business date should be the previous Sunday - To backfill a month-end task (running on January 31), the business date should be January 30 - Backfilling on a non-scheduled date will result in a dry run FILE:references/troubleshooting.md # Troubleshooting Guide This document compiles common errors encountered during DataWorks data development and their solutions, organized into validation-phase errors and API call errors. --- ## Critical: Plugin Installation Trap (READ FIRST) **Symptom**: `aliyun help` output shows `"Plugin available but not installed: aliyun-cli-dataworks-public"` and the agent installs it via `aliyun plugin install`. **Why this is WRONG**: Installing the legacy plugin activates deprecated kebab-case commands (`create-file`, `create-business`, `create-folder`). Once installed, `--help` only shows these deprecated commands, causing the agent to use the entirely wrong API set. All these deprecated APIs will fail the eval. **Correct action**: **Do NOT install any plugin.** The DataWorks 2024-05-18 APIs work via PascalCase RPC direct invocation without any plugin: ```bash # CORRECT — no plugin needed aliyun dataworks-public CreateNode --ProjectId 585549 --Scene DATAWORKS_PROJECT --Spec '...' aliyun dataworks-public CreateWorkflowDefinition --ProjectId 585549 --Spec '...' # WRONG — never do this aliyun plugin install --names aliyun-cli-dataworks-public # ← NEVER aliyun dataworks-public create-file ... # ← NEVER (kebab-case) aliyun dataworks-public create-business ... # ← NEVER (kebab-case) ``` This requires `aliyun` CLI >= 3.3.1. If the PascalCase command returns "unknown command", upgrade the CLI, do NOT install the plugin. --- ## Validation Phase Common Errors These errors are detected when running `validate.py` and must be fixed before building and submitting to the API. ### 1. command-language-match: Node Type Mismatch **Error message**: ``` ERROR: command-language-match - script.runtime.command and script.language must match the registry definition ``` **Cause**: The combination of `script.runtime.command` and `script.language` is not in the registry, or they do not match. **Common cases**: - `command` is `ODPS_SQL` but `language` is written as `sql` (correct: `odps-sql`) - `command` is `DIDE_SHELL` but `language` is written as `bash` (correct: `shell`) - A non-existent `command` value was used **Solution**: 1. Consult `assets/registry/node-types.json` to find the correct command and language mapping 2. Refer to the "Common Node Types" table in SKILL.md **Fix example**: ```json // Incorrect "script": { "language": "sql", "runtime": { "command": "ODPS_SQL" } } // Correct "script": { "language": "odps-sql", "runtime": { "command": "ODPS_SQL" } } ``` ### 2. datasource-required: Missing Datasource Configuration **Error message**: ``` ERROR: datasource-required - Node types that require a datasource must configure the datasource field ``` **Cause**: The node type has a non-null `datasourceType` in the registry, but the `datasource` field is not configured in spec.json. **Common node types that require a datasource**: - `ODPS_SQL` (requires `odps` datasource) - `HOLOGRES_SQL` (requires `hologres` datasource) - `FLINK_SQL_STREAM` / `FLINK_SQL_BATCH` (requires `flink` datasource) - `EMR_HIVE` (requires `emr` datasource) - `CLICK_SQL` (requires `clickhouse` datasource) **Solution**: Add the `datasource` field to the node definition in spec.json: ```json "datasource": { "name": "${spec.datasource.name}", "type": "odps" } ``` Also configure the actual datasource name in `dataworks.properties`: ```properties spec.datasource.name=my_odps_datasource ``` ### 3. datasource-type-match: Datasource Type Mismatch **Error message**: ``` ERROR: datasource-type-match - datasource.type must match the datasourceType for the command in the registry ``` **Cause**: The value of `datasource.type` does not match the `datasourceType` for the `command` in the registry. **Fix example**: ```json // Incorrect: HOLOGRES_SQL node using odps datasource type "script": { "runtime": { "command": "HOLOGRES_SQL" } }, "datasource": { "name": "my_ds", "type": "odps" } // Correct "script": { "runtime": { "command": "HOLOGRES_SQL" } }, "datasource": { "name": "my_ds", "type": "hologres" } ``` ### 4. code-file-exists: Missing Code File **Error message**: ``` ERROR: code-file-exists - Code file must exist and its extension must match the node type's extension in the registry ``` **Cause**: The code file is missing from the node directory, or the file extension is incorrect. **Common cases**: - Node type is `ODPS_SQL` (requires `.sql` file), but only a `.py` file was created - Node type is `DIDE_SHELL` (requires `.sh` file), but the code file is named `.bash` - Node type is `DI` (requires `.json` file), but the code file has a `.txt` extension - Forgot to create the code file **Solution**: Check the `extension` field of the node type in the registry and create a code file with the corresponding extension. | command | Correct Extension | |---------|---------| | `DIDE_SHELL` | `.sh` | | `ODPS_SQL` | `.sql` | | `PYTHON` | `.py` | | `DI` | `.json` | | `HOLOGRES_SQL` | `.sql` | | `VIRTUAL` | `.vi` | | `EMR_HIVE` | `.sql` | ### 5. properties-exists: Missing Properties File **Error message**: ``` ERROR: properties-exists - dataworks.properties file must exist ``` **Cause**: The `dataworks.properties` file is missing from the node directory. **Solution**: Create a `dataworks.properties` file in the node directory: ```properties projectIdentifier=my_project_name spec.runtimeResource.resourceGroup=S_res_group_xxx ``` If the node requires a datasource, also add: ```properties spec.datasource.name=my_datasource_name ``` ### 6. properties-no-placeholder: Properties Contains Placeholders **Error message**: ``` ERROR: properties-no-placeholder - dataworks.properties values must not contain ${...} placeholders ``` **Cause**: A value in `dataworks.properties` contains an unresolved `${...}` placeholder. The properties file is the final assignment point for placeholders; its values must be actual values. **Fix example**: ```properties # Incorrect: value contains placeholders spec.datasource.name=${datasource} spec.runtimeResource.resourceGroup=${resource_group} # Correct: use actual values spec.datasource.name=my_odps_datasource spec.runtimeResource.resourceGroup=S_res_group_524257424_1234567890 ``` ### 7. properties-key-prefix: Properties Key Prefix Error **Error message**: ``` ERROR: properties-key-prefix - dataworks.properties keys must start with spec. or script. (except projectIdentifier) ``` **Cause**: A key in `dataworks.properties` does not conform to the prefix convention. **Fix example**: ```properties # Incorrect: custom prefix datasource.name=my_ds resource_group=S_res_group_xxx # Correct: use standard prefixes spec.datasource.name=my_ds spec.runtimeResource.resourceGroup=S_res_group_xxx ``` Allowed prefixes: - `projectIdentifier` (special key, used directly) - `spec.` (for replacing placeholders in spec.json) - `script.` (for replacing placeholders in code files) ### 8. dependency-output-format: Dependency Output Format Warning **Error message**: ``` WARNING: dependency-output-format - Dependency output format should be projectIdentifier.nodeName or projectIdentifier_root ``` **Cause**: The format of `dependencies[].depends[].output` does not conform to the `projectIdentifier.nodeName` or `projectIdentifier_root` convention. **Common incorrect formats**: ```json // Incorrect: missing project identifier "output": "upstream_node" // Incorrect: using slash separator "output": "my_project/upstream_node" // Correct "output": "${projectIdentifier}.upstream_node" "output": "${projectIdentifier}_root" ``` ### 9. trigger-format: Scheduling Trigger Missing cron **Error message**: ``` ERROR: trigger-format - Scheduler-type trigger must include a cron expression ``` **Cause**: `trigger.type` is `"Scheduler"` but no `cron` expression is configured. **Fix example**: ```json // Incorrect: missing cron "trigger": { "type": "Scheduler" } // Correct "trigger": { "type": "Scheduler", "cron": "00 00 00 * * ?", "startTime": "1970-01-01 00:00:00", "endTime": "9999-01-01 00:00:00", "timezone": "Asia/Shanghai" } ``` ### 10. timeout-default: Default Timeout Warning **Error message**: ``` WARNING: timeout-default - Timeout is set to the default value of 4 hours; consider adjusting based on actual needs ``` **Description**: This is a warning, not an error, alerting that the node uses the default 4-hour timeout. For shorter tasks, consider reducing the timeout; for long-running tasks, you may need to extend it. ```json // Short task, set to 30 minutes "timeout": 30, "timeoutUnit": "MINUTES" // Long task, set to 8 hours "timeout": 8, "timeoutUnit": "HOURS" ``` --- ## API Call Common Errors These errors are returned when calling DataWorks OpenAPI. ### 0. CRITICAL Anti-Pattern: Giving Up and Saving Files Locally **Symptom**: After one or more API call failures, the agent stops calling APIs and instead saves JSON spec files to local disk, then declares the task "successfully completed" with instructions for the user to "manually create" the workflow in the UI or SDK. **Why this is WRONG**: The user asked the agent to create the workflow/node via API. Saving files locally means nothing was actually created. This is task abandonment, not completion. **Root cause**: The agent encounters an API error (usually invalid FlowSpec format) and, instead of fixing the spec, tries increasingly divergent approaches (wrapper scripts, different API structures, file-based workflows) until it gives up entirely. **Correct recovery**: 1. When an API call fails, read the error message carefully 2. Compare your spec **field by field** against the exact Quick Start example in SKILL.md 3. The most common cause is an invented field (`apiVersion`, `metadata`, `kind: "Workflow"`, `schedule`, `type`) — see the FlowSpec Anti-Patterns table 4. Copy the working example from Quick Start and modify only the values (name, content, etc.) 5. Retry with the fixed spec 6. **Only claim success when the API returns `{"Id": "..."}`** ### 0a. Invalid FlowSpec Format (Error Code 58014884415) **Error message**: ``` ErrorCode: 58014884415 ``` **Cause**: The `--Spec` JSON passed to `CreateWorkflowDefinition` or `CreateNode` has an invalid FlowSpec format. Common mistakes: - Using `"apiVersion": "v1"` instead of `"version": "2.0.0"` - Using `"kind": "Workflow"` instead of `"kind": "CycleWorkflow"` - Using `"metadata": {"name": "..."}` (not a FlowSpec field) - Missing `script.path` or `script.runtime.command` **Solution**: Fix the FlowSpec format. **Do NOT fall back to legacy APIs** (`CreateFolder`, `CreateFile`). The correct minimal FlowSpec for a workflow: ```json {"version":"2.0.0","kind":"CycleWorkflow","spec":{"workflows":[{"name":"my_workflow","script":{"path":"my_workflow","runtime":{"command":"WORKFLOW"}}}]}} ``` The correct minimal FlowSpec for a node: ```json {"version":"2.0.0","kind":"Node","spec":{"nodes":[{"name":"my_node","script":{"path":"my_node","runtime":{"command":"DIDE_SHELL"},"content":"#!/bin/bash\necho done"}}]}} ``` Refer to the FlowSpec Anti-Patterns table and Quick Start in SKILL.md for the exact format. ### 0a1. UpdateNode: "spec kind and request not match" **Error message**: ``` spec kind and request not match ``` **Cause**: You passed `"kind":"CycleWorkflow"` (or another wrong kind) in the `--Spec` of `UpdateNode`. `UpdateNode` **always** requires `"kind":"Node"`, even if the node belongs to a workflow. **Wrong**: ```json {"version":"2.0.0","kind":"CycleWorkflow","spec":{"nodes":[{"id":"NODE_ID","script":{"content":"..."}}]}} ``` **Correct**: ```json {"version":"2.0.0","kind":"Node","spec":{"nodes":[{"id":"NODE_ID","script":{"content":"new SQL here"}}]}} ``` **Do NOT fall back to `UpdateFile`** — that is a legacy API. Fix the `kind` field and retry `UpdateNode`. ### 0a2. Anti-Pattern: Creating Wrapper Scripts for API Calls **Symptom**: Agent creates a `.sh` or `.py` script file that contains multiple `aliyun` CLI commands, then executes the script. When errors occur inside the script, the agent cannot diagnose which command failed or why. **Why this is WRONG**: Wrapper scripts obscure error output, make it impossible to inspect individual API responses, and lead to cascading failures where the agent cannot determine what went wrong. **Correct approach**: Run each `aliyun` CLI command **directly** in the shell, one at a time: ```bash # Step 1: Create workflow — check response aliyun dataworks-public CreateWorkflowDefinition --ProjectId 585549 \ --Spec '{"version":"2.0.0","kind":"CycleWorkflow","spec":{"workflows":[{"name":"my_wf","script":{"path":"my_wf","runtime":{"command":"WORKFLOW"}}}]}}' \ --user-agent AlibabaCloud-Agent-Skills # → Read the response. Extract the Id. Only proceed if successful. # Step 2: Create first node — check response aliyun dataworks-public CreateNode --ProjectId 585549 --Scene DATAWORKS_PROJECT \ --ContainerId $WORKFLOW_ID \ --Spec '...' \ --user-agent AlibabaCloud-Agent-Skills # → Read the response. Only proceed if successful. ``` ### 0a3. Anti-Pattern: Using Legacy Deployment APIs (DeployFile, SubmitFile, ListDeploymentPackages) **Symptom**: Agent tries to deploy a workflow using `DeployFile`, `SubmitFile`, `ListDeploymentPackages`, `GetDeploymentPackage`, or `ListDeploymentPackageFiles`. These calls either fail with "Code does not exist" or return irrelevant results. **Why this is WRONG**: These are all legacy DataWorks APIs from older API versions. The 2024-05-18 version uses a completely different deployment model based on pipelines. **Also wrong**: Using `ListFiles` / `GetFile` to find node FileIds for deployment. These are legacy file-model APIs. Use `ListNodes` / `GetNode` / `ListWorkflowDefinitions` instead. **Correct approach**: Use the pipeline-based deployment APIs: ```bash # Step 1: Find the workflow or node ID aliyun dataworks-public ListWorkflowDefinitions --ProjectId 585549 --Type CycleWorkflow \ --user-agent AlibabaCloud-Agent-Skills # → Find the Id of the target workflow # Step 2: Create a pipeline run to deploy aliyun dataworks-public CreatePipelineRun --ProjectId 585549 \ --Type Online --ObjectIds '["WORKFLOW_ID"]' \ --user-agent AlibabaCloud-Agent-Skills # → Returns {"Id": "PIPELINE_RUN_ID"} # Step 3: Poll and advance stages aliyun dataworks-public GetPipelineRun --ProjectId 585549 --Id PIPELINE_RUN_ID \ --user-agent AlibabaCloud-Agent-Skills # → Check Pipeline.Status and Pipeline.Stages[].Status # → When Stage.Status=Init and prior stages are Success: aliyun dataworks-public ExecPipelineRunStage --ProjectId 585549 --Id PIPELINE_RUN_ID \ --Code STAGE_CODE --user-agent AlibabaCloud-Agent-Skills ``` ### 0b. Used Legacy API (Error Code 1201111431 / folder path Related Errors) **Error message**: ``` Error code: 1201111431 Message: /workflowroot/xxx or /bizroot/xxx or folder path not found ``` Or called commands like `create-file`, `create-folder`, `list-folders`, `CreateFlowProject`, etc. **Cause**: Used the legacy DataWorks API (based on the folder/business flow model). This skill uses the 2024-05-18 version OpenAPI, which does not require folder operations. **How to tell**: If you find yourself constructing paths like `/bizroot`, `/workflowroot`, or folder paths, you are on the wrong track. **Solution**: 1. **Immediately stop** folder-related operations 2. Return to the "Create Node" process in SKILL.md, using FlowSpec + `CreateNode` API 3. Use `CreateWorkflowDefinition` to create workflows, not `CreateFlowProject` / `CreateBusiness` 4. No need to install the `aliyun-cli-dataworks-public` legacy plugin **Correct API call pattern** (PascalCase RPC direct call; DataWorks 2024-05-18 has no plugin mode): ```bash # Create node (2024-05-18 version) aliyun dataworks-public CreateNode \ --ProjectId $PROJECT_ID \ --Scene DATAWORKS_PROJECT \ --Spec "$(cat /tmp/spec.json)" \ --user-agent AlibabaCloud-Agent-Skills # Create workflow (2024-05-18 version) aliyun dataworks-public CreateWorkflowDefinition \ --ProjectId $PROJECT_ID \ --Spec "$(cat /tmp/workflow_spec.json)" \ --user-agent AlibabaCloud-Agent-Skills ``` ### 1. Script path not match name **Error message**: ``` Script path must end with node name ``` **Cause**: When submitting to the API, the `script.path` field does not match the node `name`. DataWorks requires `script.path` to end with the node name. **Solution**: Ensure `script.path` ends with the `name` value, or leave `script.path` empty (let the system auto-generate it). ```json // Incorrect "name": "etl_daily", "script": { "path": "workflow/other_name" } // Correct "name": "etl_daily", "script": { "path": "workflow/etl_daily" } ``` ### 2. Spec JSON parse failed **Error message**: ``` Failed to parse Spec JSON / Invalid Spec format ``` **Cause**: - Spec JSON format is invalid (syntax error) - Missing required fields (e.g., `version`, `kind`, `spec`) - Incorrect field types **Troubleshooting steps**: 1. Check syntax with a JSON formatter (e.g., `python -m json.tool /tmp/spec.json`) 2. Confirm the three required top-level fields `version`, `kind`, `spec` are present 3. Run `validate.py` for local validation ### 3. Cannot change node type **Error message**: ``` Node type (command) cannot be changed after creation ``` **Cause**: Attempted to modify `script.runtime.command` of an existing node via the UpdateNode API. Node type is immutable after creation. **Solution**: 1. Inform the user that the node type cannot be modified after creation 2. Suggest creating a new node with the correct type and a different name 3. The user can handle the old node manually via the DataWorks console if needed ### 4. Node already exists **Error message**: ``` Node with the same name already exists in the project ``` **Cause**: A node with the same name already exists in the project. Node names are globally unique within a project. **Solution**: 1. Rename the new node (recommended) 2. If the intent is to update an existing node, use the `UpdateNode` API instead 3. Inform the user of the conflict and let them decide (rename / update existing) **Prevention**: Call `ListNodes` before creation to check if a node with the same name exists (see "Environment Awareness" in SKILL.md) ### 5. ContainerId required for workflow node **Error message**: ``` ContainerId is required when creating node in workflow ``` **Cause**: The `ContainerId` parameter was not provided when creating a node within a workflow. **Solution**: ```bash aliyun dataworks-public CreateNode \ --ProjectId {{project_id}} \ --Scene DATAWORKS_PROJECT \ --ContainerId {{workflow_id}} \ --Spec "$(cat /tmp/spec.json)" \ --user-agent AlibabaCloud-Agent-Skills ``` ### 6. Invalid cron expression **Error message**: ``` Invalid cron expression in trigger ``` **Cause**: The cron expression in `trigger.cron` has an incorrect format. **DataWorks cron format**: 6 fields (second minute hour day month weekday) **Common errors**: ``` # Incorrect: 5 fields (missing seconds) 0 0 * * ? # Correct: 6 fields 00 00 00 * * ? ``` ### 7. Resource group not found **Error message**: ``` Resource group not found or not available ``` **Cause**: The resource group identifier specified in `runtimeResource.resourceGroup` does not exist or the current project does not have access to it. **Solution**: ```bash # Query available resource groups aliyun dataworks-public ListResourceGroups \ --ProjectId {{project_id}} \ --user-agent AlibabaCloud-Agent-Skills ``` Verify the correct resource group identifier is being used. ### 8. Datasource not found **Error message**: ``` Datasource not found in project ``` **Cause**: The datasource name specified in `datasource.name` does not exist in the project. **Solution**: ```bash # Query registered datasources aliyun dataworks-public ListDataSources \ --ProjectId {{project_id}} \ --user-agent AlibabaCloud-Agent-Skills ``` Verify the datasource name is spelled correctly. ### 9. Workflow definition not found **Error message**: ``` Workflow definition not found ``` **Cause**: The specified `ContainerId` (workflow ID) does not exist. **Solution**: - Confirm the workflow was created successfully - Check that the `ContainerId` value is the correct ID returned by `CreateWorkflowDefinition` ### 10. Pipeline run failed **Error message**: ``` Pipeline run status: FAIL ``` **Cause**: The deployment pipeline failed. **Troubleshooting steps**: ```bash # Query deployment details aliyun dataworks-public GetPipelineRun \ --ProjectId {{project_id}} \ --Id {{pipeline_run_id}} \ --user-agent AlibabaCloud-Agent-Skills ``` Common failure causes: - Node code compilation errors - Dependent node does not exist - Insufficient permissions --- ## Fallback Strategy When Node Type Not Found When the node type needed by the user cannot be found in the registry, follow this strategy: ### Strategy 1: Fuzzy Matching Search for similar names in `assets/registry/node-types.json`: ```bash # Search for node types containing "hive" grep -i "hive" $SKILL/assets/registry/node-types.json ``` For example, when the user says "Hive node", it may correspond to `EMR_HIVE`, `CDH_HIVE`, etc. ### Strategy 2: Search by Category The `category` field in the registry identifies node classification: | category | Description | Typical Nodes | |----------|------|---------| | `general` | General scripts | DIDE_SHELL, PYTHON, VIRTUAL | | `maxcompute` | MaxCompute series | ODPS_SQL, ODPS_MR | | `maxcompute_resource` | MaxCompute resources | ODPS_RESOURCE, ODPS_FUNCTION | | `data_integration` | Data integration | DI | | `hologres` | Hologres series | HOLOGRES_SQL | | `flink` | Flink series | FLINK_SQL_STREAM, FLINK_SQL_BATCH | | `emr` | EMR series | EMR_HIVE, EMR_SPARK | ### Strategy 3: Fall Back to DIDE_SHELL If no matching node type can be found, use `DIDE_SHELL` (Shell script) as a universal fallback. In the Shell script, invoke the appropriate command-line tools to complete the task. ```json "script": { "language": "shell", "runtime": { "command": "DIDE_SHELL" } } ``` Shell scripts can invoke various CLI tools, covering the vast majority of scenarios. ### Strategy 4: Confirm with the User If none of the above strategies apply, clearly inform the user: 1. No exact matching node type was found in the current registry 2. List the closest candidate types 3. Suggest the user confirm the specific requirement or provide more information --- ## Quick Diagnostic Flow When encountering errors, troubleshoot in the following order: ``` 1. Run validate.py |-- Has errors -> Fix per "Validation Phase Common Errors" above +-- No errors -> Continue 2. Run build.py to build |-- Build fails -> Check spec.json syntax and properties configuration +-- Build succeeds -> Continue 3. Call the API |-- API returns error -> Troubleshoot per "API Call Common Errors" above +-- API succeeds -> Continue 4. Deploy |-- Deployment fails -> Query PipelineRun status and details +-- Deployment succeeds -> Done ``` --- ## Critical Issues Discovered Through Testing The following issues were discovered through actual API calls and are not yet clearly documented in official documentation. ### 11. Dependencies Silently Ignored **Symptom**: Dependencies were set during CreateNode, but after creation the node's dependencies are still the project root node (or none). There are three common causes: **Cause A: Upstream node did not declare `outputs.nodeOutputs`** The upstream node must declare outputs, otherwise downstream references silently fail: ```json "outputs": { "nodeOutputs": [{"data": "${projectIdentifier}.node_name", "artifactType": "NodeOutput"}] } ``` **Cause B: `nodeId` was set to the upstream node's name instead of the current node's name** `spec.dependencies[*].nodeId` is a **self-reference** — it must be the **current node's own `name`** (the node being created), NOT the upstream node's name or API-returned ID. `depends[].output` is the upstream node's output. **Cause C: `depends[].output` does not exactly match upstream's `outputs.nodeOutputs[].data`** The two values must be **character-for-character identical**. Common mismatches include wrong `projectIdentifier`, wrong node name spelling, or using dot vs underscore (`project.root` vs `project_root`). **Correct approach**: `nodeId` = current node's own name (self), `depends[].output` = upstream's output: ```json { "spec": { "nodes": [{ "name": "current_node", "id": "current_node", "outputs": { "nodeOutputs": [{"data": "${projectIdentifier}.current_node", "artifactType": "NodeOutput"}] } }], "dependencies": [{ "nodeId": "current_node", "depends": [{"type": "Normal", "output": "${projectIdentifier}.upstream_node"}] }] } } ``` See `assets/templates/05-cycle-workflow/` for a complete example. ### 11b. Deployment Fails with "can not exported multiple nodes into the same output" **Symptom**: `CreatePipelineRun` deployment fails at the PROD stage with error: `"the output name of current workspace:XXX node:YYY and that of workspace:XXX node:YYY are the same one:XXX.YYY, can not exported multiple nodes into the same output"` **Cause**: Two nodes in the same project have the same `outputs.nodeOutputs[].data` value. Output names must be **globally unique within the project**, even across different workflows. This commonly happens when recreating nodes that already exist in a different workflow. **Prevention**: Before creating any node, check for existing nodes with the same name and verify their output names: ```bash aliyun dataworks-public ListNodes --ProjectId $PID --Name "node_name" \ --user-agent AlibabaCloud-Agent-Skills ``` If a node with the same output name already exists, either: 1. Use a different node name (e.g., add a suffix) 2. Update the existing node instead of creating a new one **Recovery**: If the node was already created with a conflicting output, inform the user of the conflict and let them decide how to resolve it (rename or update existing). ### 11c. CreateNode Silently Drops spec.dependencies **Symptom**: `CreateNode` returns success, but `ListNodeDependencies` for the created node shows `TotalCount: 0` — no dependencies were persisted, despite `spec.dependencies` being correctly formatted in the request. **Cause**: The `CreateNode` API may silently discard `spec.dependencies` in certain conditions. This is a known API behavior, not a spec formatting issue. **Fix**: After creating all nodes, verify each downstream node's dependencies with `ListNodeDependencies`. If `TotalCount` is `0`, re-apply dependencies via `UpdateNode` using `spec.dependencies`: ```bash aliyun dataworks-public UpdateNode --ProjectId $PID --Id $NODE_ID \ --Spec '{"version":"2.0.0","kind":"Node","spec":{"nodes":[{"id":"'$NODE_ID'"}],"dependencies":[{"nodeId":"node_name","depends":[{"type":"Normal","output":"project.upstream_node"}]}]}}' \ --user-agent AlibabaCloud-Agent-Skills ``` **NEVER use `inputs.nodeOutputs` to fix dependencies** — always use `spec.dependencies` in the UpdateNode call. **Prevention**: Always run the "Verify and Fix Dependencies" step (see workflow-guide.md Step 5) before deploying. ### 12. datasource.type Auto-Corrected by Server **Symptom**: Submitted `datasource.type` as `flink`, but the server returned `flink_serverless`. **Explanation**: The server automatically corrects `datasource.type` based on the actual datasource type. Known corrections: - `flink` -> `flink_serverless` (Serverless Flink datasource) - Other types may have similar corrections **Handling**: Use the generic type when submitting (e.g., `flink`); no need to worry about the server-corrected actual value. ### 13. Flink Node Spec Missing dependencies Field **Symptom**: When retrieving a Flink node's spec via GetNode, the returned JSON does not contain a `dependencies` field. **Explanation**: Scheduling dependencies for Flink streaming nodes are configured via `spec.dependencies`. The spec returned by GetNode may not include the `dependencies` field; use the `ListNodeDependencies` API to query instead. ### 14. Duplicate Resource Created by Network Retry **Symptom**: A Create API call (e.g., `CreateNode`, `CreateWorkflowDefinition`) timed out or returned a network error, so the agent retried the same call. The retry succeeded, but now two identical resources exist (e.g., two nodes with the same name, or creation fails with "Node with the same name already exists"). **Cause**: The original request was actually processed by the server, but the response was lost due to a network issue. The DataWorks 2024-05-18 Create APIs do not support `ClientToken` for idempotent retries, so a blind retry creates a duplicate. **Correct recovery**: 1. **Before retrying any Create call that failed with a network/timeout error**, use the corresponding List API to check whether the resource was already created: ```bash # Example: check if node was created despite the error aliyun dataworks-public ListNodes --ProjectId $PID --Name "my_node" \ --user-agent AlibabaCloud-Agent-Skills ``` 2. If the resource exists → do NOT retry; use the existing resource's ID and continue 3. If the resource does not exist → safe to retry the Create call 4. Always record the `RequestId` from every API response for traceability **Prevention**: Always perform the pre-creation conflict check (see "Environment Discovery" in SKILL.md) before calling any Create API. This catches both pre-existing resources and resources created by prior failed attempts. ### 15. API Throttling (Throttling.User) **Error message**: ``` Code: 9990020002 Message: Throttling.User ``` **Explanation**: API call frequency exceeded the rate limit within a short period. Batch operations (such as looping GetNode to retrieve node details) are prone to triggering this. **Solution**: Add intervals between batch operations (e.g., 500ms between each call), or reduce unnecessary GetNode calls. FILE:references/verification-method.md # DataWorks Data Development Verification Methods This document describes how to verify whether DataWorks data development operations were successful. ## Node Verification ### Verify Node Creation Success ```bash # Query by node ID aliyun dataworks-public GetNode \ --ProjectId {{project_id}} \ --Id {{node_id}} \ --user-agent AlibabaCloud-Agent-Skills ``` **Success indicators**: - Returns HTTP 200 - `body.name` matches the name specified during creation - `body.script.runtime.command` matches the specified node type ### Verify Node List ```bash aliyun dataworks-public ListNodes \ --ProjectId {{project_id}} \ --Scene DATAWORKS_PROJECT \ --PageNumber 1 \ --PageSize 100 \ --user-agent AlibabaCloud-Agent-Skills ``` **Success indicators**: - The returned node list includes the newly created node - `pagingInfo.nodes[].name` matches the target node ### Verify Node Dependency Configuration ```bash # Get node details, check dependency configuration aliyun dataworks-public GetNode \ --ProjectId {{project_id}} \ --Id {{node_id}} \ --user-agent AlibabaCloud-Agent-Skills ``` **Success indicators** (parsed from the returned spec JSON): - `spec.dependencies[0].depends` contains upstream dependency relationships - The upstream node's `outputs.nodeOutputs` contains the corresponding output declaration ## Workflow Verification ### Verify Workflow Creation Success ```bash aliyun dataworks-public GetWorkflowDefinition \ --ProjectId {{project_id}} \ --Id {{workflow_id}} \ --user-agent AlibabaCloud-Agent-Skills ``` **Success indicators**: - Returns HTTP 200 - `body.name` matches the name specified during creation - `body.type` is `CycleWorkflow` or `ManualWorkflow` ### Verify Workflow List ```bash aliyun dataworks-public ListWorkflowDefinitions \ --ProjectId {{project_id}} \ --Type CycleWorkflow \ --PageNumber 1 \ --PageSize 100 \ --user-agent AlibabaCloud-Agent-Skills ``` **Success indicators**: - The returned list includes the newly created workflow ### Verify Nodes Within a Workflow ```bash # Query the node list within a workflow aliyun dataworks-public ListNodes \ --ProjectId {{project_id}} \ --Scene DATAWORKS_PROJECT \ --ContainerId {{workflow_id}} \ --PageNumber 1 \ --PageSize 100 \ --user-agent AlibabaCloud-Agent-Skills ``` **Success indicators**: - The returned node list includes all nodes created within the workflow - The node count matches expectations ## Deployment Verification ### Verify Deployment Process Creation Success ```bash aliyun dataworks-public GetPipelineRun \ --ProjectId {{project_id}} \ --Id {{pipeline_run_id}} \ --user-agent AlibabaCloud-Agent-Skills ``` **Success indicators**: - `body.pipeline.Status` is not empty - `body.pipeline.Stages` contains the deployment stage list ### Verify Final Deployment Status ```bash # Poll until deployment completes aliyun dataworks-public GetPipelineRun \ --ProjectId {{project_id}} \ --Id {{pipeline_run_id}} \ --user-agent AlibabaCloud-Agent-Skills ``` **Success indicators**: - `body.pipeline.Status` is `Success` **Failure indicators**: - `body.pipeline.Status` is `Fail`, `Termination`, or `Cancel` - Check `body.pipeline.Message` for error details ### Verify Deployment Items ```bash aliyun dataworks-public ListPipelineRunItems \ --ProjectId {{project_id}} \ --PipelineRunId {{pipeline_run_id}} \ --PageNumber 1 \ --PageSize 50 \ --user-agent AlibabaCloud-Agent-Skills ``` **Success indicators**: - All `pipelineRunItems[].Status` values indicate success - All deployed nodes/workflows are in the list ## Data Source and Resource Group Verification ### Verify Data Source Availability ```bash aliyun dataworks-public ListDataSources \ --ProjectId {{project_id}} \ --Type odps \ --PageNumber 1 \ --PageSize 100 \ --user-agent AlibabaCloud-Agent-Skills ``` **Success indicators**: - The returned data source list includes the data source name referenced in spec.json ### Verify Resource Group Availability ```bash aliyun dataworks-public ListResourceGroups \ --ProjectId {{project_id}} \ --user-agent AlibabaCloud-Agent-Skills ``` **Success indicators**: - The returned resource group list includes the resource group identifier referenced in spec.json ## Local Verification ### Verify spec.json Format ```bash $PYTHON $SKILL/scripts/validate.py ./my_node ``` **Success indicators**: - Output "Result: 0 errors, 0 warnings" - Exit code is 0 ### Verify Build Output ```bash $PYTHON $SKILL/scripts/build.py ./my_node > /tmp/spec.json python -m json.tool /tmp/spec.json > /dev/null && echo "JSON valid" ``` **Success indicators**: - Build succeeds without errors - Output JSON format is valid ## Common Verification Failure Handling | Verification Failure Scenario | Possible Cause | Solution | |-------------|---------|---------| | Node not found | Create API call failed | Check API response and error message | | Dependencies not effective | spec.dependencies not correctly configured or upstream missing outputs | Configure dependencies in spec.dependencies, ensure upstream declares outputs.nodeOutputs | | Deployment failed | Code errors or insufficient permissions | Check Stage.Message for details | | Data source not found | Name misspelled | Confirm with ListDataSources | | Resource group invalid | Identifier is wrong | Confirm with ListResourceGroups | FILE:references/workflow-guide.md # Workflow Development Guide This document describes how to create and manage workflows in DataWorks, including the complete development process for both cycle workflows and manual workflows. ## Workflow Types DataWorks supports two workflow types: ### CycleWorkflow Runs automatically according to a preset scheduling cycle, suitable for daily ETL, scheduled reports, and similar scenarios. - Must configure a `trigger` (scheduling trigger) - Nodes within the workflow execute in dependency order - FlowSpec `kind` is `"CycleWorkflow"` ### ManualWorkflow Runs only when manually triggered, suitable for data repair, one-time tasks, and similar scenarios. - No `trigger` configuration required - Must be triggered manually or via API - FlowSpec `kind` is `"ManualWorkflow"` --- ## Complete Workflow Creation Process ### Step 1: Create the Workflow Definition First, create the FlowSpec definition file for the workflow. **Cycle Workflow**: ```bash mkdir -p ./my_wf # Build the workflow spec JSON (refer to the workflow creation example in SKILL.md) ``` Edit `my_wf.spec.json`, filling in the workflow name and scheduling trigger: ```json { "version": "2.0.0", "kind": "CycleWorkflow", "spec": { "workflows": [ { "name": "my_wf", "script": { "path": "my_wf", "runtime": { "command": "WORKFLOW" } }, "trigger": { "type": "Scheduler", "cron": "00 00 00 * * ?", "startTime": "1970-01-01 00:00:00", "endTime": "9999-01-01 00:00:00", "timezone": "Asia/Shanghai" } } ] } } ``` > **`script.runtime.command: "WORKFLOW"` must be set**, otherwise `CreateWorkflowDefinition` will return an error `"script.runtime.command is empty"`. **Manual Workflow**: ```bash mkdir -p ./my_manual_wf # Build the manual workflow spec JSON (kind=ManualWorkflow) ``` Edit `my_manual_wf.spec.json`: ```json { "version": "2.0.0", "kind": "ManualWorkflow", "spec": { "workflows": [ { "name": "my_manual_wf", "script": { "path": "my_manual_wf", "runtime": { "command": "WORKFLOW" } } } ] } } ``` Build the spec JSON and call the API to create the workflow: ```bash aliyun dataworks-public CreateWorkflowDefinition \ --ProjectId {{project_id}} \ --Spec "$(cat /tmp/wf.json)" \ --user-agent AlibabaCloud-Agent-Skills ``` **Important**: Record the returned `WorkflowId`, as it is needed when creating nodes within the workflow. ### Step 2: Create Nodes Within the Workflow Each node in the workflow must be created individually. Use the `ContainerId` parameter to associate the node with the workflow. ```bash # Create node directory mkdir -p ./my_wf/step1 # Copy node template # Refer to templates in assets/templates/ and modify accordingly # [Required] Add outputs.nodeOutputs for each node (not included in minSpec): # "outputs":{"nodeOutputs":[{"data":"${projectIdentifier}.node_name","artifactType":"NodeOutput"}]} # Edit spec.json, write code file, configure properties # ...(follow the standard node creation process) # Build spec JSON and submit, note the --ContainerId parameter aliyun dataworks-public CreateNode \ --ProjectId {{project_id}} \ --Scene DATAWORKS_PROJECT \ --ContainerId {{workflow_id}} \ --Spec "$(cat /tmp/step1.json)" \ --user-agent AlibabaCloud-Agent-Skills ``` Repeat this step for each node in the workflow. ### Complete Example: 3-Node Workflow (extract -> transform -> load) Below is a complete copy-ready example showing the spec structure of nodes within a workflow. For the full version, see `assets/templates/05-cycle-workflow/`. **Root node (no upstream dependency) -- extract.spec.json**: ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "extract", "id": "extract", "script": { "path": "extract", "runtime": { "command": "DIDE_SHELL" } }, "runtimeResource": { "resourceGroup": "${spec.runtimeResource.resourceGroup}" }, "outputs": { "nodeOutputs": [ { "data": "${projectIdentifier}.extract", "artifactType": "NodeOutput" } ] } } ], "dependencies": [ { "nodeId": "extract", "depends": [ { "type": "Normal", "output": "${projectIdentifier}_root" } ] } ] } } ``` **Middle node (depends on extract) -- transform.spec.json**: ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "transform", "id": "transform", "script": { "path": "transform", "runtime": { "command": "ODPS_SQL" } }, "datasource": { "name": "${spec.datasource.name}", "type": "odps" }, "runtimeResource": { "resourceGroup": "${spec.runtimeResource.resourceGroup}" }, "outputs": { "nodeOutputs": [ { "data": "${projectIdentifier}.transform", "artifactType": "NodeOutput" } ] } } ], "dependencies": [ { "nodeId": "transform", "depends": [ { "type": "Normal", "output": "${projectIdentifier}.extract" } ] } ] } } ``` **Terminal node (depends on transform) -- load.spec.json**: ```json { "version": "2.0.0", "kind": "Node", "spec": { "nodes": [ { "name": "load", "id": "load", "script": { "path": "load", "runtime": { "command": "HOLOGRES_SQL" } }, "datasource": { "name": "${spec.datasource.name}", "type": "hologres" }, "runtimeResource": { "resourceGroup": "${spec.runtimeResource.resourceGroup}" }, "outputs": { "nodeOutputs": [ { "data": "${projectIdentifier}.load", "artifactType": "NodeOutput" } ] } } ], "dependencies": [ { "nodeId": "load", "depends": [ { "type": "Normal", "output": "${projectIdentifier}.transform" } ] } ] } } ``` **Key points**: - Each node **must** declare `outputs.nodeOutputs` (format `${projectIdentifier}.node_name`), otherwise downstream dependencies will silently fail - The output name (`${projectIdentifier}.node_name`) must be **globally unique within the project**. If another node (even in a different workflow) already uses the same output name, deployment will fail with `"can not exported multiple nodes into the same output"`. Always check with `ListNodes --Name node_name` before creating - Dependencies are configured via `spec.dependencies` only (do NOT dual-write `inputs.nodeOutputs`): - `nodeId` = the **current node's own name** (self-reference, NOT the upstream node) - `depends[].output` = the **upstream node's output** (`${projectIdentifier}.upstream_node_name`) - The upstream's `outputs.nodeOutputs[].data` and downstream's `depends[].output` must be **character-for-character identical** - Root nodes (no upstream) depend on `${projectIdentifier}_root` (underscore, not dot) - When `datasource` and `runtimeResource` are uncertain, they can be omitted; the server will automatically use the project defaults - For additional optional fields (trigger, rerunTimes, parameters, etc.), see `assets/templates/05-cycle-workflow/` ### Step 3: Configure Dependencies Dependencies between nodes within a workflow are configured via the `spec.dependencies` array only (do NOT dual-write `inputs.nodeOutputs`): > **Dependency configuration rules**: > 1. **Upstream nodes** declare `outputs.nodeOutputs`: `{"data":"${projectIdentifier}.node_name","artifactType":"NodeOutput"}` > 2. **Downstream nodes** declare dependencies in `spec.dependencies`, referencing the upstream `outputs.nodeOutputs[].data` > > **⚠️ `nodeId` is a SELF-REFERENCE** — it must be the **current node's own `name`** (the node that HAS the dependency), NOT the upstream node's name or API-returned ID. For example, if you are creating node `"step2"` and it depends on `"step1"`, then `nodeId` must be `"step2"` (self), and `depends[].output` must be `"${projectIdentifier}.step1"` (upstream's output). > > **`outputs.nodeOutputs[].data` and `dependencies[].depends[].output` must be character-for-character identical** (e.g., `${projectIdentifier}.upstream_node`); any mismatch will cause the dependency to silently fail. > > **Output names must be globally unique within the project.** Before creating any node, use `ListNodes --Name node_name` to verify the output name `${projectIdentifier}.node_name` is not already used by another node. Duplicate output names cause deployment failure: `"can not exported multiple nodes into the same output"`. > > Note: The minSpec template does not include the outputs field; it must be added manually when creating workflow nodes. **Single dependency chain** (step1 -> step2 -> step3): In step2's `spec.json`: ```json "dependencies": [ { "nodeId": "step2", "depends": [ { "type": "Normal", "output": "${projectIdentifier}.step1" } ] } ] ``` In step3's `spec.json`: ```json "dependencies": [ { "nodeId": "step3", "depends": [ { "type": "Normal", "output": "${projectIdentifier}.step2" } ] } ] ``` **Multiple dependency merge** (step1 + step2 -> step3): ```json "dependencies": [ { "nodeId": "step3", "depends": [ { "type": "Normal", "output": "${projectIdentifier}.step1" }, { "type": "Normal", "output": "${projectIdentifier}.step2" } ] } ] ``` ### Step 4: Publish and Deploy After all nodes are created, publish the workflow: ```bash aliyun dataworks-public CreatePipelineRun \ --ProjectId {{project_id}} \ --Type Online \ --ObjectIds '["{{workflow_id}}"]' \ --user-agent AlibabaCloud-Agent-Skills ``` Query the publish status: ```bash aliyun dataworks-public GetPipelineRun \ --ProjectId {{project_id}} \ --Id {{pipeline_run_id}} \ --user-agent AlibabaCloud-Agent-Skills ``` ### Step 5: Verify and Fix Dependencies (MANDATORY) **The `CreateNode` API may silently drop `spec.dependencies`.** After creating all nodes but before deploying, you MUST verify dependencies by calling `ListNodeDependencies` for each downstream node: ```bash aliyun dataworks-public ListNodeDependencies \ --ProjectId {{project_id}} \ --Id {{downstream_node_id}} \ --user-agent AlibabaCloud-Agent-Skills ``` Check the response: if `TotalCount` is `0` but the node should have upstream dependencies, **fix immediately** with `UpdateNode`: ```bash aliyun dataworks-public UpdateNode --ProjectId {{project_id}} --Id {{node_id}} \ --Spec '{"version":"2.0.0","kind":"Node","spec":{"nodes":[{"id":"{{node_id}}"}],"dependencies":[{"nodeId":"{{node_name}}","depends":[{"type":"Normal","output":"{{projectIdentifier}}.{{upstream_node_name}}"}]}]}}' \ --user-agent AlibabaCloud-Agent-Skills ``` > **NEVER use `inputs.nodeOutputs` in UpdateNode** — always use `spec.dependencies`. **Do NOT proceed to deploy until all dependencies are confirmed.** Common causes of missing dependencies: 1. `CreateNode` API silently dropped `spec.dependencies` (known behavior — fix with UpdateNode) 2. `nodeId` was set to the upstream node's name instead of the current node's own name (self-reference) 3. `depends[].output` string does not exactly match the upstream's `outputs.nodeOutputs[].data` 4. Upstream node did not declare `outputs.nodeOutputs` --- ## Dependency Configuration Details ### Intra-Workflow Dependencies Dependencies between nodes within a workflow are configured via the `spec.dependencies` array only (do NOT dual-write `inputs.nodeOutputs`). **How `spec.dependencies` wiring works** (A → B, where B depends on A): ``` Node A (upstream): Node B (downstream): name: "node_a" name: "node_b" outputs.nodeOutputs[0].data: dependencies[0].nodeId: "node_b" ← SELF (current node's own name) "${projectIdentifier}.node_a" ←──MUST MATCH──→ dependencies[0].depends[0].output: "${projectIdentifier}.node_a" ← UPSTREAM's output ``` **Upstream node** (must declare outputs): ```json "outputs": { "nodeOutputs": [ {"data": "${projectIdentifier}.upstream_node", "artifactType": "NodeOutput"} ] } ``` **Downstream node** (references upstream output in `spec.dependencies`): ```json "dependencies": [ { "nodeId": "current_node", "depends": [ { "type": "Normal", "output": "${projectIdentifier}.upstream_node" } ] } ] ``` > **Common mistake**: Setting `nodeId` to the upstream node's name or API-returned ID. `nodeId` is always the **current node's own name** — it tells the system which node in the spec this dependency entry applies to. ### Cross-Workflow Dependencies When a node in the current workflow depends on a node in another workflow, the usage is the same as long as both workflows are in the same project: ```json { "nodeId": "current_node", "depends": [ { "type": "Normal", "output": "${projectIdentifier}.other_workflow_node" } ] } ``` ### Cross-Project Dependencies When depending on a node from another project, use the upstream project's `projectIdentifier` directly (without the placeholder): ```json { "nodeId": "current_node", "depends": [ { "type": "Normal", "output": "upstream_project_name.upstream_node_name" } ] } ``` Note that the output format for cross-project dependencies is `upstream_projectIdentifier.upstream_node_name`. ### No Upstream Dependency (Attach to Project Root Node) If a node has no upstream dependency, it must be attached to the project root node: ```json { "nodeId": "first_node", "depends": [ { "type": "Normal", "output": "${projectIdentifier}_root" } ] } ``` Note: The root node output format is `projectIdentifier_root` (underscore), not `projectIdentifier.root` (dot). ### Cross-Cycle Dependencies A node depends on the previous scheduling cycle's result of itself or another node: **Self-dependency**: ```json { "nodeId": "daily_incremental", "depends": [ { "type": "CrossCycleDependsOnSelf", "output": "${projectIdentifier}.daily_incremental" } ] } ``` **Depends on another node's previous cycle**: ```json { "nodeId": "current_node", "depends": [ { "type": "CrossCycleDependsOnOther", "output": "${projectIdentifier}.other_node" } ] } ``` --- ## Workflow Directory Structure ``` my_wf/ ├── my_wf.spec.json # Workflow definition (CycleWorkflow or ManualWorkflow) ├── dataworks.properties # Workflow-level configuration ├── step1/ # Child node 1 │ ├── step1.spec.json # Node FlowSpec │ ├── step1.sh # Code file │ └── dataworks.properties # Node-level configuration ├── step2/ # Child node 2 │ ├── step2.spec.json │ ├── step2.sql │ └── dataworks.properties └── step3/ # Child node 3 ├── step3.spec.json ├── step3.py └── dataworks.properties ``` Each child node directory follows the standard node file structure (spec.json + code file + dataworks.properties). --- ## Workflow Development in Git Mode In a DataWorks Git project directory, workflow development does not require API calls. Follow these steps: 1. Create the workflow spec.json and node files 2. Configure dependencies 3. `git add` and `git commit` to submit ```bash # Commit git add ./my_wf git commit -m "Add workflow: my_wf with step1, step2, step3" ``` --- ## Important Notes 1. **Creation order**: The workflow definition must be created first to obtain the returned WorkflowId before creating nodes within the workflow 2. **`script.runtime.command` is required**: The workflow spec must include `script.runtime.command: "WORKFLOW"`, otherwise `CreateWorkflowDefinition` will return an error `"script.runtime.command is empty"` 3. **ContainerId parameter**: Nodes within a workflow are associated to the workflow via the `ContainerId` parameter of the `CreateNode` API (value is the WorkflowId), rather than embedding node definitions inside the workflow spec 4. **Intra-workflow node dependencies are configured via `spec.dependencies`** only (do NOT dual-write `inputs.nodeOutputs`). `spec.dependencies[*].nodeId` is a **self-reference** — it must be the **current node's own `name`**, NOT the upstream node's name or API-returned ID. `depends[].output` is the **upstream node's output identifier** (see the complete example and template `assets/templates/05-cycle-workflow/`) 5. **Node outputs must be declared**: Each node within a workflow **must** declare `${projectIdentifier}.node_name` in `outputs.nodeOutputs` (minSpec does not include this field; it must be added manually). **Output names must be globally unique within the project** — if another node already uses the same output name, deployment will fail 6. **Root node dependency**: Nodes with no upstream dependency must be attached to `${projectIdentifier}_root` 7. **Workflow trigger**: The trigger of a cycle workflow defines the workflow-level scheduling cycle; nodes within the workflow inherit this schedule 8. **Immutable properties**: The workflow type (CycleWorkflow / ManualWorkflow) cannot be changed after creation 9. **Updates must be incremental**: When calling `UpdateNode`, only pass the id + fields to be modified; do not pass unchanged fields like `datasource` or `runtimeResource`. The server may have corrected these field values (e.g., `flink` to `flink_serverless`), and passing back the original values will cause errors 10. **GetNode returns the full workflow spec**: When calling `GetNode` on a node within a workflow, the response returns a complete `kind: CycleWorkflow` workflow spec; the target node is located at `spec.workflows[0].nodes[]` (not `spec.nodes[]`) 11. **Workflow `strategy` field**: Workflows support a `strategy` configuration (including priority, timeout, rerunMode, failureStrategy, etc.); this field is defined at the workflow level, not the node level 12. **`datasource.type` auto-correction**: The server may automatically correct the value of `datasource.type` (e.g., `flink` is corrected to `flink_serverless`); refer to the actual returned value FILE:scripts/build.py #!/usr/bin/env python3 """Merge the 3-file node/workflow structure into API-ready FlowSpec JSON. 3-file structure: my_node/ ├── my_node.spec.json # FlowSpec definition (with ${spec.xxx} placeholders) ├── my_node.sql # Code file └── dataworks.properties # Actual values for placeholders Merge logic: 1. Read dataworks.properties -> parse into key=value dict 2. Read spec.json -> replace all ${spec.xxx} and ${projectIdentifier} placeholders 3. Read code file -> embed into spec.nodes[0].script.content 4. Output the merged JSON (can be used directly as the Spec parameter for CreateNode API) Usage: python build.py ./my_node # Output to stdout python build.py ./my_node -o /tmp/spec.json # Output to file """ import json import re import sys from pathlib import Path SKIP_EXTENSIONS = {'.json', '.properties'} def parse_properties(filepath): """Parse dataworks.properties into a key=value dict.""" props = {} for line in filepath.read_text(encoding='utf-8').splitlines(): line = line.strip() if not line or line.startswith('#'): continue if '=' not in line: continue key, value = line.split('=', 1) props[key.strip()] = value.strip() return props def replace_placeholders(text, props): """Replace ${spec.xxx}, ${projectIdentifier} and other placeholders. Only replaces placeholders that have a corresponding key in properties; unmatched ones are kept as-is. """ def replacer(match): key = match.group(1) if key in props: return props[key] # Key in spec.xxx format if key.startswith('spec.') and key in props: return props[key] return match.group(0) # Keep as-is return re.sub(r'\$\{([^}]+)\}', replacer, text) def find_spec_file(node_dir): """Find the .spec.json file.""" specs = list(node_dir.glob('*.spec.json')) if len(specs) == 1: return specs[0] if len(specs) == 0: print(f"Error: No .spec.json file found in {node_dir}", file=sys.stderr) sys.exit(1) print(f"Error: Multiple .spec.json files found in {node_dir}: {[s.name for s in specs]}", file=sys.stderr) sys.exit(1) def find_code_file(node_dir): """Find the code file (excluding .spec.json and .properties).""" for f in node_dir.iterdir(): if f.is_file() and f.suffix not in SKIP_EXTENSIONS: return f return None def build(node_dir): """Merge 3-file structure into complete FlowSpec JSON.""" node_dir = Path(node_dir) if not node_dir.is_dir(): print(f"Error: {node_dir} is not a directory", file=sys.stderr) sys.exit(1) # 1. Read properties props_file = node_dir / 'dataworks.properties' props = parse_properties(props_file) if props_file.exists() else {} # 2. Read spec.json + replace placeholders spec_file = find_spec_file(node_dir) spec_text = spec_file.read_text(encoding='utf-8') spec_text = replace_placeholders(spec_text, props) spec = json.loads(spec_text) # 3. Find code file + embed into script.content code_file = find_code_file(node_dir) if code_file: code_content = code_file.read_text(encoding='utf-8') # Node: spec.nodes[0].script.content nodes = spec.get('spec', {}).get('nodes', []) if nodes: nodes[0].setdefault('script', {})['content'] = code_content # Workflows have no code file, skip return spec def main(): import argparse parser = argparse.ArgumentParser(description='Merge 3-file structure into FlowSpec JSON') parser.add_argument('dir', help='Path to node or workflow directory') parser.add_argument('-o', '--output', help='Output file path (default: stdout)') args = parser.parse_args() result = build(args.dir) output = json.dumps(result, ensure_ascii=False, indent=2) if args.output: Path(args.output).write_text(output, encoding='utf-8') print(f"Output written to {args.output}", file=sys.stderr) else: print(output) if __name__ == '__main__': main() FILE:scripts/requirements.txt # Optional: enables JSON Schema validation in validate.py jsonschema==4.23.0 FILE:scripts/validate.py #!/usr/bin/env python3 """FlowSpec Validation Tool Validates DataWorks node/workflow directories or spec.json files. Dependency: jsonschema>=4.0,<5.0 (optional, for JSON Schema validation) Usage: python validate.py # Validate a directory or file python validate.py --json # Output in JSON format """ import argparse import json import os import re import sys from pathlib import Path try: import jsonschema except ImportError: jsonschema = None # Skill root directory (relative to this script's location) SKILL_ROOT = Path(__file__).resolve().parent.parent SCHEMAS_DIR = SKILL_ROOT / "assets" / "schemas" class ValidationResult: def __init__(self, path): self.path = str(path) self.errors = [] self.warnings = [] def error(self, field, message, fix=None): entry = {"field": field, "message": message} if fix: entry["fix"] = fix self.errors.append(entry) def warning(self, field, message): self.warnings.append({"field": field, "message": message}) @property def valid(self): return len(self.errors) == 0 def to_dict(self): return { "valid": self.valid, "path": self.path, "errors": self.errors, "warnings": self.warnings, } def to_text(self): lines = [f"Validation: {self.path}", ""] for e in self.errors: lines.append(f" ❌ {e['field']}: {e['message']}") if "fix" in e: lines.append(f" Fix: {e['fix']}") for w in self.warnings: lines.append(f" ⚠️ {w['field']}: {w['message']}") lines.append("") lines.append(f"Result: {len(self.errors)} error(s), {len(self.warnings)} warning(s)") return "\n".join(lines) def load_json_schema(kind): """Load FlowSpec JSON Schema""" schema_map = { "Node": "Node.schema.json", "CycleWorkflow": "CycleWorkflow.schema.json", "ManualWorkflow": "ManualWorkflow.schema.json", } filename = schema_map.get(kind) if not filename: return None schema_path = SCHEMAS_DIR / "flowspec" / filename if not schema_path.exists(): return None with open(schema_path, "r", encoding="utf-8") as f: return json.load(f) def parse_properties(props_path): """Parse dataworks.properties file""" props = {} if not props_path.exists(): return None with open(props_path, "r", encoding="utf-8") as f: for line in f: line = line.strip() if not line or line.startswith("#"): continue if "=" in line: key, _, value = line.partition("=") props[key.strip()] = value.strip() return props def find_spec_file(directory): """Find *.spec.json file in the directory""" spec_files = list(directory.glob("*.spec.json")) if len(spec_files) == 1: return spec_files[0] if len(spec_files) > 1: # Prefer the one matching the directory name dir_name = directory.name for sf in spec_files: if sf.stem.replace(".spec", "") == dir_name: return sf return spec_files[0] return None def validate_spec_json(spec_path, result): """Validate a spec.json file""" # 1. Read and parse JSON try: with open(spec_path, "r", encoding="utf-8") as f: spec_data = json.load(f) except json.JSONDecodeError as e: result.error("spec.json", f"JSON parse failed: {e}", "Check that the JSON format is correct") return None # 2. JSON Schema validation kind = spec_data.get("kind") if not kind: result.error("kind", "Missing kind field", 'Add "kind": "Node" or "CycleWorkflow" or "ManualWorkflow"') return None if jsonschema: schema = load_json_schema(kind) if schema: try: jsonschema.validate(spec_data, schema) except jsonschema.ValidationError as e: path_str = ".".join(str(p) for p in e.absolute_path) if e.absolute_path else "root" result.error(f"schema({path_str})", str(e.message)[:200]) return spec_data def validate_node(spec_data, spec_path, directory, result): """Validate a Node type spec""" if not spec_data or spec_data.get("kind") != "Node": return nodes = spec_data.get("spec", {}).get("nodes", []) if not nodes: result.error("spec.nodes", "Node list is empty", "Define at least one node") return for i, node in enumerate(nodes): prefix = f"spec.nodes[{i}]" # Timeout default value warning timeout = node.get("timeout") timeout_unit = node.get("timeoutUnit", "HOURS") if timeout == 4 and timeout_unit == "HOURS": result.warning( f"{prefix}.timeout", "Timeout is set to the default value of 4 hours", ) # Trigger format check trigger = node.get("trigger") if trigger: if trigger.get("type") == "Scheduler" and not trigger.get("cron"): result.error( f"{prefix}.trigger.cron", "Scheduler type trigger is missing a cron expression", 'Add "cron": "00 00 00 * * ?"', ) def validate_workflow(spec_data, result): """Validate a Workflow type spec""" kind = spec_data.get("kind", "") if kind not in ("CycleWorkflow", "ManualWorkflow"): return workflows = spec_data.get("spec", {}).get("workflows", []) if not workflows: result.error("spec.workflows", "Workflow list is empty", "Define at least one workflow") return for i, wf in enumerate(workflows): prefix = f"spec.workflows[{i}]" if not wf.get("name"): result.error(f"{prefix}.name", "Workflow name cannot be empty") if kind == "CycleWorkflow": trigger = wf.get("trigger") if trigger and trigger.get("type") == "Scheduler" and not trigger.get("cron"): result.error( f"{prefix}.trigger.cron", "Cycle workflow Scheduler trigger is missing a cron expression", ) def validate_properties(directory, result): """Validate dataworks.properties""" props_path = directory / "dataworks.properties" # 8. properties exists if not props_path.exists(): result.error( "dataworks.properties", "dataworks.properties file does not exist", f"Create file {props_path}", ) return props = parse_properties(props_path) if props is None: return placeholder_pattern = re.compile(r"\$\{[^}]+\}") valid_prefixes = ("spec.", "script.", "projectIdentifier") for key, value in props.items(): # 9. No placeholders in value if placeholder_pattern.search(value): result.error( f"dataworks.properties[{key}]", f'Value contains placeholder: "{value}"', f"Replace the value of {key} with an actual value", ) # 10. Key prefix convention if not any(key.startswith(p) for p in valid_prefixes): result.error( f"dataworks.properties[{key}]", f'Key has non-standard prefix: "{key}"', 'Key must start with "spec." or "script." (except "projectIdentifier")', ) def validate_directory(directory, result, recursive=True): """Validate a node/workflow directory""" directory = Path(directory) # Find spec.json spec_file = find_spec_file(directory) if not spec_file: result.error("*.spec.json", "No .spec.json file found in directory", "Create a .spec.json file") return # Validate spec.json spec_data = validate_spec_json(spec_file, result) if not spec_data: return kind = spec_data.get("kind", "") if kind == "Node": validate_node(spec_data, spec_file, directory, result) validate_properties(directory, result) elif kind in ("CycleWorkflow", "ManualWorkflow"): validate_workflow(spec_data, result) validate_properties(directory, result) # Recursively validate child node directories if recursive: for subdir in sorted(directory.iterdir()): if subdir.is_dir() and find_spec_file(subdir): sub_result = ValidationResult(subdir) validate_directory(subdir, sub_result, recursive=False) result.errors.extend(sub_result.errors) result.warnings.extend(sub_result.warnings) def validate_file(file_path, result): """Validate a single spec.json file""" spec_data = validate_spec_json(file_path, result) if not spec_data: return kind = spec_data.get("kind", "") directory = Path(file_path).parent if kind == "Node": validate_node(spec_data, file_path, directory, result) elif kind in ("CycleWorkflow", "ManualWorkflow"): validate_workflow(spec_data, result) def main(): parser = argparse.ArgumentParser(description="FlowSpec Validation Tool") parser.add_argument("path", help="Path to directory or .spec.json file to validate") parser.add_argument("--json", action="store_true", help="Output in JSON format") args = parser.parse_args() target = Path(args.path).resolve() result = ValidationResult(args.path) if target.is_dir(): validate_directory(target, result) elif target.is_file(): validate_file(target, result) # Also check properties in the same directory if target.suffix == ".json": validate_properties(target.parent, result) else: result.error("path", f"Path does not exist: {args.path}") # Output if args.json: print(json.dumps(result.to_dict(), ensure_ascii=False, indent=2)) else: print(result.to_text()) sys.exit(0 if result.valid else 1) if __name__ == "__main__": main()

Alibabacloud Dataworks Datastudio Develop

Comments (0)