DataViewDataSource

DataViewDataSource is a federated virtual data source that unifies entities from multiple underlying datasources into a single logical view. It implements IDataSource, IDataViewOperations, and IDisposable, and persists its definition as a JSON file via DMDataView.

The view can operate in two execution modes: Cached (data is materialised into a local in-memory engine and refreshed on a TTL) or DirectQuery (always fetched live from the backing sources). Cross-source joins are expressed as FederatedJoinDefinition records and are either detected automatically from foreign-key metadata or defined manually.

Architecture

A DataViewDataSource instance owns one IDMDataView (concrete type DMDataView). The view holds the list of EntityStructure objects (each referring back to a real datasource via DataSourceID) and a list of FederatedJoinDefinition records describing how entities are joined cross-source.

DataViewDataSource (IDataSource + IDataViewOperations)
├── IDMDataView (DMDataView) — persisted as JSON
│   ├── List<EntityStructure>       — entities from 1..N datasources
│   ├── List<FederatedJoinDefinition> — cross-source joins
│   ├── ExecutionMode               — Cached | DirectQuery
│   ├── CacheTTLSeconds             — default 300 s
│   ├── LocalEngineConnectionName   — in-memory engine name
│   └── ViewName / ViewID / VID
│
├── DataViewConnection (IDataConnection, IsLogical=true)
│   └── No physical file; OpenConnection() always returns Open
│
└── Partial classes
    ├── DataViewDataSource.EntityLifecycle.cs
    ├── DataViewDataSource.FieldManagement.cs
    ├── DataViewDataSource.Filters.cs
    ├── DataViewDataSource.JoinGraph.cs
    ├── DataViewDataSource.QueryExecution.cs
    ├── DataViewDataSource.Validation.cs
    └── DataViewDataSource.ViewUtilities.cs

Execution flow — Cached mode

1. GetEntity() detects cross-source entities (multiple distinct DataSourceID values).
2. RefreshCacheIfExpired() checks CacheLastRefresh + CacheTTLSeconds.
3. If stale, PrepareMergedQueryDB() materialises all entities into the LocalEngineConnectionName in-memory store.
4. The local engine then satisfies the query — including joins — as a native SQL operation.

Execution flow — DirectQuery mode / single-source

1. Single-source entities are delegated directly to the backing IDataSource.
2. File, URL, and query view types are passed through unchanged.

Key types

DataViewConnection

DataViewConnection implements IDataConnection for view-oriented sources. Because views are logical constructs, IsLogical = true means OpenConnection() immediately returns Open without touching the filesystem.

// DataViewConnection.OpenConnection()
public ConnectionState OpenConnection()
{
    return OpenConn(); // fast-path when IsLogical=true → sets status=Open immediately
}

Core Properties

Constructor

public DataViewDataSource(
    string        datasourcename,
    IDMLogger     logger,
    IDMEEditor    pDMEEditor,
    DataSourceType pDatasourceType,
    IErrorsInfo   per);

The constructor performs the following steps:

1. Assigns Logger, ErrorObject, DMEEditor.
2. Sets DatasourceType = DataSourceType.Text, Category = DatasourceCategory.VIEWS.
3. Creates a DataViewConnection with IsLogical = true.
4. Resolves the file path from the configured FolderFileTypes.DataView folder, or from the path embedded in datasourcename.
5. Looks up an existing ConnectionProperties entry for the file name. If none exists, creates a new entry with DriverName = "DataViewReader" and saves the connections config.
6. Sets DataViewFile to the full absolute path of the JSON view definition.

Data Retrieval

Smart routing in GetEntity

// Cross-source path (entities have different DataSourceIDs at root level)
bool isCrossSource = DataView.Entities
    .Where(e => e.ParentId == 0)
    .Select(e => e.DataSourceID)
    .Distinct(StringComparer.OrdinalIgnoreCase)
    .Count() > 1;

if (isCrossSource && ent.Viewtype == ViewType.Table)
{
    RefreshCacheIfExpired();
    var db = PrepareMergedQueryDB();
    return db?.GetEntity(EntityName, filter);
}

// Single-source fallback
IDataSource srcDs = GetDataSourceObject(EntityName);
return srcDs?.GetEntity(EntityName, filter);

Entity Lifecycle

Defined in DataViewDataSource.EntityLifecycle.cs. All mutating methods call InvalidateCache() automatically.

Field Management

Defined in DataViewDataSource.FieldManagement.cs.

// Add a computed/virtual field to a view entity
var field = new EntityField
{
    FieldName  = "FullName",
    fieldtype  = "System.String",
    IsKey      = false,
    CustomField = true
};
var result = view.AddFieldToEntity("CUSTOMERS", field);

// Sync schema from live source after upstream changes
var changes = view.RefreshEntitySchema("ORDERS");
if (changes.Any())
    Console.WriteLine("Schema drift: " + string.Join(", ", changes));

Filter Management

Defined in DataViewDataSource.Filters.cs. Filters are stored in EntityStructure.CustomBuildQuery property (a WHERE-clause expression per entity). Setting or clearing a filter automatically calls InvalidateCache().

// Apply a WHERE filter to one entity in the view
view.SetEntityFilter("ORDERS", "Status = 'ACTIVE' AND Amount > 1000");

// Read it back
string expr = view.GetEntityFilter("ORDERS");

// Remove just that entity's filter
view.ClearEntityFilter("ORDERS");

// Wipe all filters in one call
view.ClearAllFilters();

Join Graph Intelligence

Defined in DataViewDataSource.JoinGraph.cs. Provides automatic FK-based join detection, BFS path-finding, and a graph adjacency export.

FederatedJoinDefinition properties

// Auto-detect joins from FK metadata
view.AutoDetectJoins();

// Find shortest join path between two entities
var path = view.FindJoinPath("CUSTOMERS", "ORDERITEMS");
// → ["CUSTOMERS", "ORDERS", "ORDERITEMS"]

// Inspect the graph
var graph = view.GetJoinGraph();
foreach (var kvp in graph)
    Console.WriteLine($"{kvp.Key} → [{string.Join(", ", kvp.Value)}]");

// Manually define a cross-source join
view.JoinDefinitions.Add(new FederatedJoinDefinition
{
    GuidID            = Guid.NewGuid().ToString(),
    LeftEntityName    = "CUSTOMERS",
    LeftColumn        = "CustomerID",
    LeftDataSourceID  = "SqliteDB",
    RightEntityName   = "CRMENTRIES",
    RightColumn       = "CustID",
    RightDataSourceID = "CrmRestApi",
    JoinType          = FederatedJoinType.Left,
    IsManuallyDefined = true
});

Query Execution & Cache

Defined in DataViewDataSource.QueryExecution.cs.

// Check cache state before deciding whether to force refresh
var (isExpired, _, lastRefresh, remaining) = view.GetCacheStatus();
Console.WriteLine($"Cache expires in {remaining}s (last: {lastRefresh})");

// Force async re-materialise (e.g. after a batch ETL)
await view.MaterializeAsync(cts.Token);

// Print the execution plan for debugging
Console.WriteLine(view.GetExecutionPlan());
// -- DataView: SalesView
// -- Execution Mode: Cached
// -- Entities: 3, Joins: 2
// SELECT * FROM [CUSTOMERS]
// INNER JOIN [ORDERS] ON [CUSTOMERS].[CustomerID] = [ORDERS].[CustomerID]
// INNER JOIN [ORDERITEMS] ON [ORDERS].[OrderID] = [ORDERITEMS].[OrderID]

Cache invalidation

InvalidateCache() is called automatically by any mutation that changes the view structure: adding/removing entities, changing filters, moving entities, and modifying joins. It resets CacheLastRefresh to DateTime.MinValue so that the next GetEntity() call triggers a fresh materialise.

Validation & Health

Defined in DataViewDataSource.Validation.cs. All methods are non-destructive — they open connections only to test reachability, never modify data.

// Full view validation
var errors = view.ValidateView();
if (errors.Any())
{
    foreach (var e in errors) Console.WriteLine("ERROR: " + e);
}

// Identify entities that can't be reached
var broken = view.GetBrokenEntities();
Console.WriteLine($"Broken: {string.Join(", ", broken)}");

// Find isolated entities (no joins to rest of the graph)
var lone = view.GetUnconnectedEntities();
Console.WriteLine($"Unconnected: {string.Join(", ", lone)}");

// Check for upstream schema drift on one entity
var drift = view.DetectSchemaChanges("PRODUCTS");
if (drift.Any())
    Console.WriteLine("Schema changed: " + string.Join("; ", drift));

View Utilities

Defined in DataViewDataSource.ViewUtilities.cs.

ViewSummary properties

// Get a full health snapshot
var summary = view.GetViewSummary();
Console.WriteLine($"Valid: {summary.IsValid}");
Console.WriteLine($"Entities: {summary.EntityCount}, Joins: {summary.JoinCount}");
Console.WriteLine($"Sources: {summary.DataSourceCount}");
Console.WriteLine($"Cache age: {(DateTime.UtcNow - summary.CacheLastRefresh).TotalSeconds:F0}s");

// Clone for dev/test environment
var devView = (DMDataView)view.CloneView("SalesView_DEV");

// Merge a partner team's view definition
var partnerView = JsonSerializer.Deserialize<DMDataView>(File.ReadAllText("partner.json"));
view.MergeView(partnerView);

Examples

Example 1 — Create a federated view from two sources

// 1. Resolve the DataViewDataSource from DMEEditor
var view = (DataViewDataSource)dme.GetDataSource("SalesView.json");

// 2. Load the view definition
view.GetEntitesList();

// 3. Verify entities and auto-detect joins from FK metadata
view.AssignDataSourceToEntities(); // ensure DataSourceIDs are set
view.AutoDetectJoins();

// 4. Show the execution plan
Console.WriteLine(view.GetExecutionPlan());

// 5. Execute a cross-source query
var orders = view.GetEntity("ORDERS", new List<AppFilter>
{
    new AppFilter { FieldName = "Status", Operator = "=", FilterValue = "OPEN" }
});

Example 2 — Apply filters and check cache

// Narrow each participating entity before materialising
view.SetEntityFilter("ORDERS",    "OrderDate > '2025-01-01'");
view.SetEntityFilter("CUSTOMERS", "Region = 'EMEA'");

// Cache was just invalidated. Force async materialise.
await view.MaterializeAsync();

// Check that the cache is now warm
var (expired, _, refreshed, ttlLeft) = view.GetCacheStatus();
Console.WriteLine($"Cache fresh — {ttlLeft}s remaining (refreshed {refreshed:u})");

// Query now routes through the local engine
var rows = view.GetEntity("ORDERS", null);
Console.WriteLine($"Rows: {rows?.Count()}");

Example 3 — View health check and schema sync

var summary = view.GetViewSummary();

if (!summary.IsValid)
{
    Console.WriteLine("Validation errors:");
    summary.Warnings.ForEach(w => Console.WriteLine("  " + w));
}

if (summary.BrokenEntities.Any())
{
    Console.WriteLine("Broken entities (check datasource connections):");
    summary.BrokenEntities.ForEach(b => Console.WriteLine("  " + b));
}

if (summary.UnconnectedEntities.Any())
{
    Console.WriteLine("Isolated entities (no join path):");
    summary.UnconnectedEntities.ForEach(u => Console.WriteLine("  " + u));
}

// Sync schema for any entity whose upstream definition may have changed
foreach (string entityName in view.EntitiesNames)
{
    var drift = view.DetectSchemaChanges(entityName);
    if (drift.Any())
    {
        Console.WriteLine($"Schema drift on {entityName} — refreshing...");
        var changes = view.RefreshEntitySchema(entityName);
        changes.ForEach(c => Console.WriteLine("  Applied: " + c));
    }
}

Example 4 — Clone and merge views

// Create a test copy of the production view
var devView = view.CloneView("SalesView_Test");
// devView is a fully independent DMDataView — saving it will not affect the original

// Merge a partner view (e.g. Inventory entities) into the current view
var inventoryDef = JsonSerializer.Deserialize<DMDataView>(
    File.ReadAllText("InventoryView.json"));

view.MergeView(inventoryDef);

// Add a manual cross-datasource join for the newly merged entities
view.JoinDefinitions.Add(new FederatedJoinDefinition
{
    GuidID            = Guid.NewGuid().ToString(),
    LeftEntityName    = "ORDERS",
    LeftColumn        = "ProductID",
    LeftDataSourceID  = "SalesDB",
    RightEntityName   = "PRODUCTS",
    RightColumn       = "ProductID",
    RightDataSourceID = "InventoryDB",
    JoinType          = FederatedJoinType.Left,
    IsManuallyDefined = true,
    Description       = "Sales ↔ Inventory product link"
});

Console.WriteLine($"View now has {view.JoinDefinitions.Count} joins across {view.GetAllDataSourceIDs().Count} sources.");

Full API Reference — DataViewDataSource

Core Properties — Complete (17 + inherited)

Complete Method Index by Partial File

DataViewDataSource.cs — Core

// Constructor
public DataViewDataSource(IDataConnection connection, IDMDataView dmDef,
    IDMEEditor editor, string name = null, string viewName = null);

// IDataSource passthrough (Open/Close delegate to Dataconnection)
public ConnectionState Openconnection();
public ConnectionState Closeconnection();

EntityLifecycle.cs

public IEnumerable<object> GetEntity(string entityName, List<AppFilter> filters);
public IErrorsInfo InsertEntity(string entityName, object inserted);
public IErrorsInfo UpdateEntity(string entityName, object updated);
public IErrorsInfo DeleteEntity(string entityName, object deleted);
public IErrorsInfo UpdateEntities(string entityName, object data, IProgress<PassedArgs> progress = null);

FieldManagement.cs

public void AddField(DMDataViewField field);
public void RemoveField(string guidID);
public void ClearFields();
public void UpdateField(DMDataViewField field);
public bool AreAnyFieldsBroken();
public List<DMDataViewField> GetBrokenFieldsMap();
public List<DMDataViewField> GetVisibleFields();
public string ResolveColumnAlias(string entityName, string columnName);
public string ResolveEntityNameForBroken(string brokenFieldGuidID);

Filters.cs

public void AddFilter(DMFilterCondition filter);
public void RemoveFilter(string id);
public void ClearFilters();
public void UpdateFilter(DMFilterCondition filter);
public List<DMFilterCondition> GetAllFilters();
public List<DMFilterCondition> GetFilterByEntity(string entityName);
public bool AreAnyFiltersBroken();
public List<DMFilterCondition> GetBrokenFiltersMap();
public List<AppFilter> BuildAppFilterList();

JoinGraph.cs

public void AddJoin(FederatedJoinDefinition join);
public void RemoveJoin(string guidID);
public void ClearJoins();
public void UpdateJoin(FederatedJoinDefinition join);
public List<FederatedJoinDefinition> GetJoins();
public bool AutoDetectJoins();  // Uses FK metadata from entity structures
public List<FederatedJoinDefinition> FindJoinPath(string fromEntity, string toEntity);  // BFS
public void BuildJoinGraph();   // Adjacency list from JoinDefinitions
public bool IsJoinGraphConnected();

QueryExecution.cs

public Task<DataTable> ExecuteViewQueryAsync(string targetEntity,
    List<AppFilter> filters = null, CancellationToken ct = default);
public IErrorsInfo RunScript(string scriptName, object script);

Validation.cs

public bool ValidateView();
public void RefreshHealth();
public ViewHealthReport GetViewHealthReport();
public event EventHandler<ViewHealthChangedEventArgs> ViewHealthChanged;

ViewUtilities.cs

public void Save(string filePath);
public void Load(string filePath);
public DMDataView CloneView(string newViewName = null);
public void MergeView(DMDataView otherView);
public void ResetCache();
public bool IsCacheExpired();
public Dictionary<string, object> GetViewMetadata();

DataViewConnection — Complete

public class DataViewConnection : IDataConnection
{
    // Properties
    public bool InMemory { get; set; }
    public bool IsLogical { get; set; }
    public IConnectionProperties ConnectionProp { get; set; }
    public ConnectionDriversConfig DataSourceDriver { get; set; }
    public ConnectionState ConnectionStatus { get; set; }
    public IDMEEditor DMEEditor { get; set; }
    public int ID { get; set; }
    public string GuidID { get; set; } = Guid.NewGuid().ToString();
    public IDMLogger Logger { get; set; }
    public IErrorsInfo ErrorObject { get; set; }
    public IDbConnection DbConn { get; set; }

    // Methods
    public ConnectionState OpenConnection();
    public ConnectionState CloseConn();

    // Logical connection support
    private ConnectionState OpenConn();  // IsLogical skips filesystem check
}

Source File Map (9 files)