RDBMSDataSource Architecture
Internal design of RDBSource — the core RDBMS plugin with 16 partial class files, 5 helper classes, and support for 8+ database engines.
Class Hierarchy
IRDBSource
Framework interface defining the RDBMS contract: CRUD, schema loading, query execution, DDL generation, and transaction management.
RDBSource
Main implementation — a partial class split across 16 files organized by responsibility. ~5,400 lines total. Database-agnostic via ADO.NET abstraction.
InMemoryRDBSource
Extends RDBSource with IInMemoryDB. Adds ETL-based data sync, lifecycle management (IsCreated, IsLoaded, IsSynced), and structure creation.
RDBDataConnection
Wraps IDbConnection. Handles driver instantiation via DMEEditor.assemblyHandler, connection string building, schema alteration.
Partial Class Breakdown (16 Files)
| File | Lines | Responsibility |
|---|---|---|
RDBSource.cs | 164 | Core properties, constructor, EntityStructureCache, events |
RDBSource.Connection.cs | 33 | Open/close delegation to RDBDataConnection |
RDBSource.Transaction.cs | 66 | Begin/End/Commit transaction (⚠️ Commit stub) |
RDBSource.CRUD.cs | 585 | InsertEntity, UpdateEntity, DeleteEntity + async variants |
RDBSource.Query.cs | 1,004 | GetEntity, RunQuery, ExecuteSql, BuildQuery, SQL parsing |
RDBSource.DMLGeneration.cs | 911 | INSERT/UPDATE/DELETE SQL generation, DDL, parameter creation |
RDBSource.Schema.cs | 830 | Entity structure loading, FK detection, Oracle FLOAT mapping |
RDBSource.BulkOperations.cs | 1,080 | Bulk insert/update, temp-table strategies, multi-row optimization |
RDBSource.Cache.cs | 252 | Query cache, result cache (MemoryCache), prepared statements |
RDBSource.Modernization.cs | 497 | IAsyncEnumerable streaming, async paging, scalar/non-query |
RDBSource.Resilience.cs | 426 | Polly v8 retry + circuit breaker, health checks |
RDBSource.Dapper.cs | 38 | Dapper Query<T> + ExecuteAsync<T> wrappers |
RDBSource.TypeMapping.cs | 108 | DbType conversion + value conversion (all .NET types) |
RDBSource.Utilities.cs | 297 | Error handling, GetDataCommand, GetDataAdapter, naming |
RDBSource.Pagination.cs | 12 | ⚠️ Empty stub — paging logic lives in Modernization.cs |
RDBSource.Dispose.cs | 45 | IDisposable pattern, connection cleanup |
Helper Classes
| Class | Lines | Purpose |
|---|---|---|
DataStreamer | 109 | Streams IDataReader → Dictionary<string,object> or typed objects with cached property mapping |
DbTypeMapper | 38 | Static dictionary mapping .NET type strings → DbType |
EntityStructureCache | 27 | Thread-safe ConcurrentDictionary with refresh-on-demand loader |
PagedQueryExecutor | 118 | Builds count query + paged query, executes both, returns (rows, total) |
PaginationHelper | 15 | Delegates to RDBMSHelper.GetPagingSyntax for DB-specific paging |
Key Design Patterns
Provider Abstraction
Database drivers are loaded dynamically via DMEEditor.assemblyHandler.GetInstance(typeName). The plugin has no compile-time dependency on any specific ADO.NET provider. Supported engines:
| Engine | Paging Syntax | Temp Tables | Health Check |
|---|---|---|---|
| SQL Server | OFFSET/FETCH | #temp | SELECT 1 |
| MySQL | LIMIT/OFFSET | CREATE TEMPORARY TABLE | SELECT 1 |
| PostgreSQL | LIMIT/OFFSET | CREATE TEMP TABLE | SELECT 1 |
| Oracle | ROWNUM subquery | — | SELECT 1 FROM DUAL |
| SQLite | LIMIT/OFFSET | — | SELECT 1 |
| DB2 | — | — | SELECT 1 FROM SYSIBM.SYSDUMMY1 |
Resilience Pipeline
Polly v8 ResiliencePipeline combines circuit breaker (50% failure ratio, 30s duration) + retry (exponential backoff, 3 attempts). Health checks issue database-specific ping queries and auto-reopen connections.
Multi-Level Caching
- Query cache —
ConcurrentDictionary<string,string>for compiled query strings - Result cache —
MemoryCachewith configurable TTL (default 5 min, size limit 1024 entries) - Prepared statement cache —
ConcurrentDictionaryfor command text reuse - Entity structure cache —
EntityStructureCache(ConcurrentDictionary with refresh) - Property mapping cache —
DataStreamercaches reflected property info per type
Bulk Operation Strategies
- Multi-row INSERT —
INSERT INTO t VALUES (...), (...), (...)for SQL Server/MySQL/PostgreSQL/SQLite - Temp-table MERGE — Create temp table → bulk insert → MERGE/UPDATE JOIN for SQL Server/MySQL/PostgreSQL
- Batched fallback — Individual INSERT/UPDATE wrapped in transaction for Oracle/DB2
Known Issues
The authoritative defect register lives with the code, at
DataSourcesPlugins/RDBMSDataSource/docs/10-known-issues.md, where every
entry carries a reproducing scenario and a fix direction. The table below is a
severity summary only.
| Severity | Issue | Location |
|---|---|---|
| 🔴 Critical | A null Logger makes AddLogMessage return before setting the error flag, so ~40 failure paths report success — including a CREATE TABLE that never ran | BeepDM DMEEditor.cs:204 |
| 🔴 Critical | UPDATE primary-key parameters are resolved by substring, so an UPDATE can target a row chosen by the wrong column's value | DMLGeneration.cs:304 |
| 🔴 Critical | Zero rows affected is logged as a failure but returned as success; bulk success counters inherit this | CRUD.cs:74 |
| 🔴 Critical | GetFieldName emits a string literal rather than a quoted identifier on every driver but SQLite; reserved-word columns are never quoted at all | Utilities.cs:272 |
| 🔴 Critical | SQL injection through AppFilter.Operator and FieldName — filter values are parameterised, these are not | BeepDM DataSourceAppFilterExtensions.cs |
| 🔴 Critical | Bulk insert reflects on typeof(T), so IEnumerable<object> input writes an all-NULL table and reports success | BulkOperations.cs:763 |
| 🔴 Critical | Every SQL Server connection runs ALTER LOGIN … DEFAULT_DATABASE, a permanent server-wide change to the login | RDBDataConnection.cs:180 |
| 🟠 High | Hana, Snowflake and Presto set ParameterDelimiter = "?", so their write path cannot bind at all | driver constructors |
| 🟠 High | Bulk batches open a transaction the row commands never join — Oracle bulk insert fails by default | BulkOperations.cs:312 |
| 🟠 High | Temp-table DDL emits .NET type names as SQL types, so bulk update fails on SQL Server, MySQL and PostgreSQL | BulkOperations.cs:950 |
| 🟠 High | Schema and table concatenated with no . separator at six sites, breaking bulk and the async/paged read paths | BulkOperations.cs, Modernization.cs |
| 🟠 High | Two paging implementations with opposite defaults; AzureSQL and SqlCompact produce a syntax error on every paged read | Query.cs:797, Modernization.cs:412 |
| 🟠 High | GetTableName rewrites SQL by string surgery; an entity named updates throws IndexOutOfRangeException | Utilities.cs:291 |
| 🟡 Medium | The Polly resilience layer is wired to no query path, and would not fire anyway because OpenConn swallows exceptions internally | Resilience.cs |
| 🟡 Medium | Most of Cache.cs has no callers yet logs “Cache invalidated” on every write; EntityStructureCache, the live one, has no invalidation at all | Cache.cs, EntityStructureCache.cs |
| 🟡 Medium | Per-operation state is held in shared instance fields; the “thread-safe per-operation” comment is inverted | RDBSource.cs:32 |
| 🟡 Medium | Dispose nulls its collections and releases neither the active transaction, the retained command, nor the connection | Dispose.cs |
Previously listed here and since fixed:
UpdateEntities() calling InsertEntity();
Commit() and EndTransaction() as empty stubs (commit c3901629);
the MemoryCache key-enumeration gap.
Developer reference
A per-file walkthrough of all sixteen partial classes lives alongside the source, at
DataSourcesPlugins/RDBMSDataSource/docs/.
| Document | Covers |
|---|---|
README.md | File map, the fourteen drivers, what a driver actually overrides |
01-error-model.md | How success and failure are reported — read first |
02-lifecycle.md | Construction, connection, shared state, disposal |
03-transactions.md | _activeTransaction, and why EndTransaction means rollback |
04-crud-dml.md | The write path, parameter binding, DDL generation |
05-query-paging.md | The read path and its four paging routes |
06-schema-types.md | Structure discovery, entity types, type mapping |
07-bulk.md | Batching, temp tables, transaction scope |
08-cache-resilience.md | What is live, and what is built but not wired up |
09-extending.md | Writing a driver: what to override, what to avoid |
10-known-issues.md | The defect register with reproducing scenarios |