ETL family: sources, transforms, lookups and destinations

Overview

Base Control: ETLControl : MaterialControl
Namespace: Beep.Skia.ETL

The ETLControl is the root control for this diagram family. It inherits MaterialControl and provides shared port management, styling, and template methods for all nodes in the family.

This family contains 18 node types.

Node Types

Node ClassShapeDescription
ETLSourceTable/source iconData extraction source (database, file, API).
ETLTransformGear/rectangleGeneric data transformation step.
ETLTargetDatabase/target iconData loading destination.
ETLDestinationTarget containerAlternative destination node.
ETLFilterFunnelRow-level filtering by condition.
ETLSortArrow up/downSort operation on data stream.
ETLJoinMerge iconJoin two data streams (inner/left/right/full).
ETLMergeMerge (vertical)Merge/union compatible data streams.
ETLLookupMagnifying glassReference data lookup/join.
ETLAggregateSigma symbolAggregation (SUM, COUNT, AVG, GROUP BY).
ETLPivotArrow rotatePivot rows to columns.
ETLUnpivotArrow counter-rotateUnpivot columns to rows.
ETLConditionalSplitDiamondRoute rows to different outputs based on conditions.
ETLDerivedColumnColumn iconCompute new column from expression.
ETLMulticastTriangle splitDuplicate stream to multiple outputs.
ETLRowCountCounterCount rows passing through.
ETLScdClock iconSlowly Changing Dimension (Type 1/2/3).
ETLScriptCode blockCustom script/expression execution.

ETLSource

Data extraction source (database, file, API).

ShapeTable/source icon
Input Ports0
Output Ports1+

ETLTransform

Generic data transformation step.

ShapeGear/rectangle
Input Ports1+
Output Ports1+

ETLTarget

Data loading destination.

ShapeDatabase/target icon
Input Ports1+
Output Ports0-1

ETLDestination

Alternative destination node.

ShapeTarget container
Input Ports1+
Output Ports0

ETLFilter

Row-level filtering by condition.

ShapeFunnel
Input Ports1
Output Ports1

ETLSort

Sort operation on data stream.

ShapeArrow up/down
Input Ports1
Output Ports1

ETLJoin

Join two data streams (inner/left/right/full).

ShapeMerge icon
Input Ports2
Output Ports1

ETLMerge

Merge/union compatible data streams.

ShapeMerge (vertical)
Input Ports2+
Output Ports1

ETLLookup

Reference data lookup/join.

ShapeMagnifying glass
Input Ports2
Output Ports1

ETLAggregate

Aggregation (SUM, COUNT, AVG, GROUP BY).

ShapeSigma symbol
Input Ports1
Output Ports1

ETLPivot

Pivot rows to columns.

ShapeArrow rotate
Input Ports1
Output Ports1

ETLUnpivot

Unpivot columns to rows.

ShapeArrow counter-rotate
Input Ports1
Output Ports1

ETLConditionalSplit

Route rows to different outputs based on conditions.

ShapeDiamond
Input Ports1
Output Ports2+

ETLDerivedColumn

Compute new column from expression.

ShapeColumn icon
Input Ports1
Output Ports1

ETLMulticast

Duplicate stream to multiple outputs.

ShapeTriangle split
Input Ports1
Output Ports2+

ETLRowCount

Count rows passing through.

ShapeCounter
Input Ports1
Output Ports1

ETLScd

Slowly Changing Dimension (Type 1/2/3).

ShapeClock icon
Input Ports1
Output Ports1

ETLScript

Custom script/expression execution.

ShapeCode block
Input Ports1+
Output Ports1+

Connection Points

Every node in this family exposes connection points (ports) for linking to other nodes. Port positions are computed lazily by LayoutPorts(), which runs when MarkPortsDirty() flags the layout as stale after a move or resize.

Node ClassInput PortsOutput PortsExtra
ETLSource01+—
ETLTransform1+1+—
ETLTarget1+0-1—
ETLDestination1+0—
ETLFilter11—
ETLSort11—
ETLJoin21—
ETLMerge2+1—
ETLLookup21—
ETLAggregate11—
ETLPivot11—
ETLUnpivot11—
ETLConditionalSplit12+—
ETLDerivedColumn11—
ETLMulticast12+—
ETLRowCount11—
ETLScd11—
ETLScript1+1+—

Connection Conventions

When building diagrams with this family, follow these conventions:

  • Flow direction: left to right or top to bottom
  • Sources have no input ports, targets have no output ports
  • Transforms may have multiple inputs/outputs
  • Line colors indicate data flow, error flow, or logging flow

Usage Example

Complete example showing creation, node placement, and connection:

C# Example

var etl = new ETLControl
{
    X = 50, Y = 50,
    Width = 800, Height = 500
};
drawingManager.AddComponent(etl);

var src = new ETLSource { X = 100, Y = 250, Text = "SQL Server" };
var filter = new ETLFilter { X = 280, Y = 250, Text = "Filter Active" };
var transform = new ETLTransform { X = 460, Y = 250, Text = "Normalize" };
var tgt = new ETLTarget { X = 640, Y = 250, Text = "Data Warehouse" };

drawingManager.AddComponent(src);
drawingManager.AddComponent(filter);
drawingManager.AddComponent(transform);
drawingManager.AddComponent(tgt);

drawingManager.ConnectComponents(src.OutPoints[0], filter.InPoints[0]);
drawingManager.ConnectComponents(filter.OutPoints[0], transform.InPoints[0]);
drawingManager.ConnectComponents(transform.OutPoints[0], tgt.InPoints[0]);