ETL family

Complete API surface of the Beep.Skia.ETL assembly: 38 public types with their documented members. Follow the family guide for usage patterns.

Types in this assembly

CdcMethod enum
ColumnProfile class
DataPreviewFormatter class
DataProfile class
DataProfiler class
DestinationKind enum
ETLAggregate class
ETLCdcNode class
ETLConditionalSplit class
ETLControl class
ETLDerivedColumn class
ETLDestination class
ETLFilter class
ETLFuzzyLookupNode class
ETLJoin class
ETLLookup class
ETLMerge class
ETLMulticast class
ETLPivot class
ETLRowCount class
ETLScd class
ETLScript class
ETLSort class
ETLSource class
ETLSurrogateKeyNode class
ETLTarget class
ETLTransform class
ETLUnpivot class
ExpressionEngine class
ExpressionException class
FlattenOptions class
FuzzyAlgorithm enum
NodeRunMetrics class
PipelineMetrics class
SourceKind enum
StructuredDataFlattener class
TransformKind enum
WriteMode enum

ColumnProfile (class)

Statistical profile of a single column.

Member	Kind	Description
`string Name`	Property	Gets or sets the name.
`int Count`	Property	Gets or sets the count.
`int NullCount`	Property	Gets or sets the null count.
`int DistinctCount`	Property	Gets or sets the distinct count.
`object Min`	Property	Gets or sets the min.
`object Max`	Property	Gets or sets the max.
`double? Average`	Property	Gets or sets the average.
`double? Sum`	Property	Gets or sets the sum.
`int? MinLength`	Property	Gets or sets the min length.
`int? MaxLength`	Property	Gets or sets the max length.
`List<object> SampleValues`	Property	Gets or sets the sample values.
`bool IsNumeric`	Property	Gets or sets the is numeric.
`bool IsDateTime`	Property	Gets or sets the is date time.
`double NullPercentage(int rowCount)`	Method	Gets or sets the null percentage.

DataPreviewFormatter (class)

Formats rows (dictionaries) as an aligned plain-text table for data previews.

Member	Kind	Description
`static string ToTable( IEnumerable<IDictionary<string, object>> rows, int maxRows = 20, int maxColumnWidth = 24)`	Method	Renders up to rows as a fixed-width text table. Long cell values are truncated to .

DataProfile (class)

Profile of a dataset: row count and per-column statistics.

Member	Kind	Description
`int RowCount`	Property	Gets or sets the row count.
`List<ColumnProfile> Columns`	Property	Gets or sets the columns.
`string Summary()`	Method	Human-readable profile report.

DataProfiler (class)

Computes column-level statistics (counts, nulls, distinct, min/max, average, string lengths) over a set of rows represented as dictionaries.

Member	Kind	Description
`DataProfile Profile(IEnumerable<IDictionary<string, object>> rows, int sampleSize = 5)`	Method	Profiles the given rows. Column order follows first-seen order.

ETLAggregate (class)

ETL Aggregate component that represents data aggregation operations in ETL workflows. Displays as an octagon shape indicating data summarization/grouping.

Member	Kind	Description
`ETLAggregate()`	Method	Initializes a new instance of the ETLAggregate class.

ETLCdcNode (class)

Member	Kind	Description
`CdcMethod Method`	Property	Gets or sets the method.
`string TrackingColumn`	Property	Gets or sets the tracking column.
`string ChangeIndicator`	Property	Gets or sets the change indicator.
`ETLCdcNode()`	Method	Initializes a new instance of the ETLCdcNode class.

ETL Conditional Split component that routes rows to multiple outputs based on conditions. Displays as a diamond with multiple output ports. Evaluates conditions in order; first match wins. Unmatched rows go to default output.

Member	Kind	Description
`string Conditions`	Property	Split conditions (JSON array of SplitCondition)
`bool HasDefaultOutput`	Property	Include default output for unmatched rows
`ETLConditionalSplit()`	Method	Initializes a new instance of the ETLConditionalSplit class.

ETLControl (class)

Base visual for ETL nodes: rounded rectangle with a header/title and optional subtitle. Provides helpers to create and maintain input/output connection points along the left/right edges.

Member	Kind	Description
`string Title`	Property	Gets or sets the title.
`bool HasErrorOutput`	Property	Enable error output port for rejected rows
`int InPortCount`	Property	Gets or sets the in port count.
`int OutPortCount`	Property	Gets or sets the out port count.

ETLDerivedColumn (class)

ETL Derived Column component for adding calculated columns to the data flow. Displays as a rectangle with expression builder support. Supports multiple derived columns with expressions using built-in function library.

Member	Kind	Description
`string DerivedColumns`	Property	Derived columns (JSON array of DerivedColumnDefinition)
`string OutputSchema`	Property	Output schema (JSON array of ColumnDefinition)
`ETLDerivedColumn()`	Method	Initializes a new instance of the ETLDerivedColumn class.

ETLDestination (class)

ETL Destination component that represents data destinations in ETL workflows. Displays as a funnel shape indicating data output/storage.

Member	Kind	Description
`ETLDestination()`	Method	Initializes a new instance of the ETLDestination class.

ETLFilter (class)

ETL Filter component that represents data filtering operations in ETL workflows. Displays as a diamond shape indicating conditional data filtering.

Member	Kind	Description
`ETLFilter()`	Method	Initializes a new instance of the ETLFilter class.

ETLFuzzyLookupNode (class)

Member	Kind	Description
`FuzzyAlgorithm Algorithm`	Property	Gets or sets the algorithm.
`double Threshold`	Property	Gets or sets the threshold.
`string MatchColumn`	Property	Gets or sets the match column.
`string ReferenceColumn`	Property	Gets or sets the reference column.
`ETLFuzzyLookupNode()`	Method	Initializes a new instance of the ETLFuzzyLookupNode class.

ETLJoin (class)

ETL Join component that represents data joining operations in ETL workflows. Displays as a triangle shape indicating data merging/combining.

Member	Kind	Description
`ETLJoin()`	Method	Initializes a new instance of the ETLJoin class.

ETLLookup (class)

ETL Lookup component that enriches data by joining with reference data. Displays as a cylinder with two inputs (main data + reference data). Supports caching strategies for performance optimization.

Member	Kind	Description
`CacheMode CacheMode`	Property	Cache strategy: None, Partial, Full
`string MatchColumns`	Property	Columns to match (CSV: MainCol=RefCol,...)
`string ReturnColumns`	Property	Columns to return from reference (CSV)
`bool FailOnNoMatch`	Property	Fail if lookup finds no match
`string OutputSchema`	Property	Output schema (JSON array of ColumnDefinition)
`ETLLookup()`	Method	Initializes a new instance of the ETLLookup class.

ETLMerge (class)

ETL Merge node: combines multiple input streams into one output (UNION ALL). Different from Join - simply concatenates rows from multiple sources.

Member	Kind	Description
`int InputCount`	Property	Gets or sets the input count.
`bool RemoveDuplicates`	Property	Gets or sets the remove duplicates.
`ETLMerge()`	Method	Initializes a new instance of the ETLMerge class.

ETLMulticast (class)

ETL Multicast node: sends identical copy of data to multiple outputs. Used for parallel processing or feeding multiple destinations.

Member	Kind	Description
`int OutputCount`	Property	Gets or sets the output count.
`ETLMulticast()`	Method	Initializes a new instance of the ETLMulticast class.

ETLPivot (class)

ETL Pivot node: rotates data from rows to columns (crosstab transformation). Converts row-based data into column-based aggregated view.

Member	Kind	Description
`string PivotColumn`	Property	Gets or sets the pivot column.
`string GroupByColumns`	Property	Gets or sets the group by columns.
`string ValueColumn`	Property	Gets or sets the value column.
`string AggregateFunction`	Property	Gets or sets the aggregate function.
`ETLPivot()`	Method	Initializes a new instance of the ETLPivot class.

ETLRowCount (class)

ETL Row Count node: counts rows passing through and optionally adds count column. Useful for auditing and debugging pipeline data flow.

Member	Kind	Description
`bool AddCountColumn`	Property	Gets or sets the add count column.
`string CountColumnName`	Property	Gets or sets the count column name.
`ETLRowCount()`	Method	Initializes a new instance of the ETLRowCount class.

ETLScd (class)

ETL SCD (Slowly Changing Dimension) component for managing dimension updates. Displays as a special cylinder with historical tracking icon. Supports Type 1 (overwrite) and Type 2 (historical with effective dates).

Member	Kind	Description
`ScdType Type`	Property	SCD Type: Type1 (overwrite), Type2 (historical)
`string BusinessKeys`	Property	Business key columns (CSV) for matching existing records
`string ChangeTrackingColumns`	Property	Columns to track for changes (CSV)
`string EffectiveFromColumn`	Property	Column name for effective from date (Type 2 only)
`string EffectiveToColumn`	Property	Column name for effective to date (Type 2 only)
`string CurrentFlagColumn`	Property	Column name for current flag (Type 2 only)
`string TableName`	Property	Target dimension table name
`ETLScd()`	Method	Initializes a new instance of the ETLScd class.

ETLScript (class)

ETL Script node: executes custom transformation script (SQL, Python, C#, etc). Allows arbitrary transformations beyond built-in operations.

Member	Kind	Description
`string ScriptLanguage`	Property	Gets or sets the script language.
`string Script`	Property	Gets or sets the script.
`ETLScript()`	Method	Initializes a new instance of the ETLScript class.

ETLSort (class)

ETL Sort node: orders rows by specified columns and direction.

Member	Kind	Description
`string SortColumns`	Property	Gets or sets the sort columns.
`string SortDirections`	Property	Gets or sets the sort directions.
`bool RemoveDuplicates`	Property	Gets or sets the remove duplicates.
`ETLSort()`	Method	Initializes a new instance of the ETLSort class.

ETLSource (class)

ETL Source component that represents data sources in ETL workflows. Displays as a cylinder shape indicating data storage/retrieval.

Member	Kind	Description
`SourceKind Kind`	Property	Source kind
`string ConnectionString`	Property	Source connection (masked)
`string Path`	Property	Table/File/Endpoint path
`ETLSource()`	Method	Initializes a new instance of the ETLSource class.

ETLSurrogateKeyNode (class)

Member	Kind	Description
`string KeyColumn`	Property	Gets or sets the key column.
`long StartValue`	Property	Gets or sets the start value.
`long Increment`	Property	Gets or sets the increment.
`ETLSurrogateKeyNode()`	Method	Initializes a new instance of the ETLSurrogateKeyNode class.

ETLTarget (class)

ETL Target (Sink) node representing a destination like a table, file, or topic. Accepts one input, validates incoming schema vs ExpectedSchema, and shows warnings on mismatch.

Member	Kind	Description
`string ConnectionString`	Property	Destination connection string (masked in UI)
`DestinationKind Destination`	Property	Destination kind
`string TableName`	Property	Target table or object name
`WriteMode Mode`	Property	Append/Overwrite/Upsert
`bool PreCreateTable`	Property	Auto-create table from expected schema
`string ExpectedSchema`	Property	Expected schema (JSON array of ColumnDefinition)
`string DataQualityRules`	Property	Data quality validation rules (JSON array of DataQualityRule)
`ETLTarget()`	Method	Initializes a new instance of the ETLTarget class.

ETLTransform (class)

ETL Transform node with basic operations: Select/Project, Filter, Map/Compute. Exposes OutputSchema (JSON of ColumnDefinition) for downstream nodes and lines.

Member	Kind	Description
`TransformKind Kind`	Property	Transform kind
`string Expression`	Property	Filter/Map expression (simple syntax)
`string OutputSchema`	Property	Output schema (JSON array of ColumnDefinition)
`string GroupBy`	Property	Aggregate group-by columns (CSV)
`string JoinKeyLeft`	Property	Join key in left input
`string JoinKeyRight`	Property	Join key in right input
`ETLTransform()`	Method	Initializes a new instance of the ETLTransform class.
`void InferOutputSchemaFromUpstreams(Func<int, string> getInputSchemaJson)`	Method	Helper used by external orchestrators or editor buttons to infer OutputSchema from upstream schemas for Aggregate and Join kinds. This is a lightweight heuristic.

ETLUnpivot (class)

ETL Unpivot node: rotates data from columns to rows (denormalization). Converts column-based data into row-based normalized format.

Member	Kind	Description
`string UnpivotColumns`	Property	Gets or sets the unpivot columns.
`string AttributeColumn`	Property	Gets or sets the attribute column.
`string ValueColumn`	Property	Gets or sets the value column.
`ETLUnpivot()`	Method	Initializes a new instance of the ETLUnpivot class.

ExpressionEngine (class)

Small SQL-like expression engine for ETL derived columns, conditional splits, filters, and custom data-quality rules. Supports: literals (number/string/TRUE/FALSE/NULL), column references ([Column], "Column", `Column`, or bare identifiers), arithmetic (+ - * / %), comparisons (= == != <> < <= > >=), logical operators (AND OR NOT), IS NULL / IS NOT NULL, LIKE, and a built-in function library. Unknown columns evaluate to null.

Member	Kind	Description
`static IReadOnlyCollection<string> Functions`	Property	Gets the names of all built-in functions.
`object Evaluate(string expression, IDictionary<string, object> row)`	Method	Evaluates an expression against a row. Returns null when the result is NULL.
`bool EvaluateBoolean(string expression, IDictionary<string, object> row)`	Method	Evaluates an expression as a boolean (null/0/empty/"false" are false).
`IReadOnlyCollection<string> GetReferencedColumns(string expression)`	Method	Returns the column names referenced by the expression (excluding function names).

ExpressionException (class)

Raised when an expression cannot be parsed or evaluated.

Member	Kind	Description
`ExpressionException(string message) : base(message) { }`	Method	Initializes a new instance of the ExpressionException class.

FlattenOptions (class)

Options controlling flattening behavior.

Member	Kind	Description
`string Separator`	Property	Separator used between nested field names (default ".").
`int MaxDepth`	Property	Maximum nesting depth to flatten.
`bool IncludeNulls`	Property	When true, JSON nulls are emitted as null values; otherwise omitted.

FuzzyAlgorithm (enum)

NodeRunMetrics (class)

Runtime metrics for a single pipeline node.

Member	Kind	Description
`string NodeName`	Property	Gets or sets the node name.
`int RowsIn`	Property	Gets or sets the rows in.
`int RowsOut`	Property	Gets or sets the rows out.
`TimeSpan Elapsed`	Property	Gets or sets the elapsed.
`string Status`	Property	Gets or sets the status.
`DateTime RecordedAt`	Property	Gets or sets the recorded at.
`double RowsPerSecond`	Property	Gets or sets the rows per second.

PipelineMetrics (class)

Collects per-node runtime metrics for an ETL pipeline run.

Member	Kind	Description
`List<NodeRunMetrics> Nodes`	Property	Gets or sets the nodes.
`DateTime? StartedAt`	Property	Gets or sets the started at.
`DateTime? CompletedAt`	Property	Gets or sets the completed at.
`bool IsRunning`	Property	Gets or sets the is running.
`TimeSpan TotalElapsed`	Property	Gets or sets the total elapsed.
`void Start()`	Method	Marks the start of a pipeline run.
`void Complete()`	Method	Marks the end of a pipeline run.
`NodeRunMetrics Record(string nodeName, int rowsIn, int rowsOut, TimeSpan elapsed, string status = "Completed")`	Method	Records a completed node execution.
`int Measure(string nodeName, int rowsIn, Func<int> action)`	Method	Measures a node action and records its metrics. Returns the action's row count.
`void Reset()`	Method	Clears all metrics and timing.
`string Summary()`	Method	Human-readable metrics report.

SourceKind (enum)

StructuredDataFlattener (class)

Flattens structured JSON/XML into tabular rows (dictionaries) for ETL processing, and infers a tabular schema from flattened rows. Nested objects become dot-separated columns (address.city); arrays are preserved as JSON text so no data is lost; XML attributes become "@name" columns.

Member	Kind	Description
`List<Dictionary<string, object>> FlattenJson(string json, string rootPath = null, FlattenOptions options = null)`	Method	Flattens a JSON document into rows. JSON text. Optional dot path to the array/object to flatten (e.g., "data.items"). Flattening options.
`List<Dictionary<string, object>> FlattenXml(string xml, string rowElement = null, FlattenOptions options = null)`	Method	Flattens an XML document into rows. When is null, the most frequent child element of the root is used as the row element.
`List<ColumnDefinition> InferSchema(IEnumerable<IDictionary<string, object>> rows, string stringType = "VARCHAR")`	Method	Infers a tabular schema from flattened rows. Numeric columns merge to FLOAT, mixed types fall back to STRING, and sparse/missing values mark the column nullable.