Release Notes And Downloads

Release date: 2023-11-14

Fixes

• Mapping handling
  • Now finds more mapping errors
  • Fixed an issue when source table for hashing procedure could not be correctly identified, causing a crash
  • Overall better handling of “dummy” columns
• Hashing
  • Fixed issue when _orig column was incorrectly handled when the incoming value had an ad hoc transformation
• Synapse
  • Now correctly extracts table metadata from Synapse source/target database
• Deployment package
  • Fixed issue where work table was not recreated in a multi-database solution

Release date: 2023-09-22

Fixes

• Fixed a bug where the D♯ Engine client would sometimes create a different UID not previously registered on the License Server, leading to a failed license retrieval
• HTML documentation
  • In Class documentation, the Natural key components are bolded and italicised in the Properties table
  • Textual documentation is now formatted as paragraphs instead of continuous text
• Fixed crash that happened if D♯ Engine was started but no license was selected in several minutes
• Deployment package
  • When deploying classes, the number of non-implemented classes to be deployed is reported to the user
  • If no implemented classes are deployed, the Deployment package will not be created at all

Release date: 2023-06-30

Features

• License service
  • Licenses now stored on and queried from DSharp’s License Server
  • Moving licenses between developers now easier
  • Same D♯ Engine installation can be used with multiple licenses/targets 
  • With no internet connection available, the license information from the previous succeeded connection remains valid for a limited time
  • Support for full offline license file
• New software installer
  • Preserves existing settings if upgrading on top of existing installation
  • Supports having parallel different versions for use with different customers

Release date: 2023-03-31

Features

• Modeling
  • Support for prioritising sources when selecting attribute values in the class view
  • Omit ghost row option has been added to Class-to-class mappings. Setting this to true filters out the ghost row from the incoming data set defined by the mapping.
• Source data mapping & hashing
  • More mapping errors found from the source mappings
    • Source mapping error messages have also been improved
  • Support for automatically loading only the newest version from a data set containing multiple versions of the same thing
  • Faster load process: minimised number of left-joins during hashing
    • Procedure code now documents the reason for including or excluding a join. This information can also be used for debugging mappings during development if a join should happen but it doesn’t, or vice versa.
    • Check your mapping files! The better you indicate primary and foreign keys, the smarter choices D♯ Engine will be able to make.
• Commands
  • New: Load versioned source data in batches as a one-off task
  • New: Delete the last load batch from the DW (“Risen Delete”)
  • New: “Re-materialize” views; runs the procedure that populates the table
  • The SQL code for dropping the object has been added to the De-Installation candidates command
• UI changes and additions
  • Technology preview: AI-based Natural Language input in the Data Grid
    • Ask D♯ Engine to implement queries, fact or dimension tables based on the selected classes
    • Try it out and go wild!
  • New Languages pane which shows all the available target platforms’ datatypes as well as all custom transformations available for that platform
  • Class mappings: new window that shows the source mappings for both Raw Model and Business Model classes
  • The SQL content of the selected items in a Deployment package are now shown in the SQL view
  • Class and property names in the UI now directly indicate whether there are issues with that class/property (for example, unmapped properties)
  • UI commands have been rearranged. As usual, Help -> Command reference displays all available commands as well as their position in the main menu.
  • Telerik UI component upgrade
• Deployment
  • The Deployment package may now include recommendations on how to finalise the deployment manually, if needed
  • When deploying objects, the user is given the possibility to also include objects that the original objects depend on
    • For example, when deploying a BO class, you may also want to deploy its source classes, because their Materialisation state may have changed
• Course material
  • Advanced course tutorial models and scripts have been added to the installation for self-studying
    • Material available here
  • The course demo data database (also included in the installation) has been redesigned and expanded
  • Visual Paradigm models now available for all individual tutorial steps in both the Intro course and the Advanced course

Fixes

• Soft delete fixed in cases where a hubless class was marked for Soft delete. In such a case the view would always behave like Soft delete was turned on, automatically leading to what in most cases was the desired functionality even without explicitly having to set Soft delete on.
• Removed an unnecessary join from the Class views
• The deployment of the LogErrors procedure was removed from the Deployment package
• Automatic state table updates leading to rows falsely being marked as deleted now removed from the Reload last failed rows command
• Generated join aliases in the hashing procedures renamed in such a way that they don’t end up as reserved words by mistake
• Parameter name “Don’t mark as deleted” changed to “Ignore when setting deleted state”
  • Update your ETLParameters table accordingly (see the Upgrade to 4.1.0 command from Help -> Upgrades.)
• Other minor fixes

There are changes in Visual Paradigm Stereotypes (SEStereotypes.xml), so you will need to re-import the file.

If you are upgrading from an earlier version, check the Help.Upgrades menu, and run each of the Upgrade commands that have a version number that is bigger than the one you are upgrading from. These commands analyse the currently active model for issues that you may need to look at.

Release date: 2023-01-10

Maintenance release.

• Updated all tutorial folders to now include a full Visual Paradigm model (plus model export) for that tutorial step with support for the latest features
• Fixed error that caused HTML documentation generation to fail
• Fixed error that prevented a FK-referenced unmapped ID column from being copied to the work table, if it was part of an aliased mapping group

If you are upgrading from an earlier version, check the Help.Upgrades menu, and run each of the Upgrade commands that have a version number that is bigger than the one you are upgrading from. These commands analyse the currently active model for issues that you may need to look at. Also make sure that you import the latest SEStereotypes.xml file into Visual Paradigm, if needed.

Release date: 2022-12-30

Features

• Synapse support, complete with automatic table distribution logic and indexing model
• Extended source data mapping functionality radically reduces the need for making manual OUT_ views
  • Map the same source table to multiple classes
    • Good for normalizing denormalized source data and splitting data that clearly contains different sets of information
    • Stronger supports the model-oriented approach in that it is easier to not by default follow the source data’s possibly suboptimal structure and to actually design the Data Vault
  • Map several source tables within the same schema to the same class
    • Replaces the need for OUT_ views performing union operations to merge several source tables into one class
  • Map the same source column to multiple properties in same class
    • “Copy” or create derived data from the same source column to another attribute
  • Create “dummy” columns for special cases
    • Create additional metadata to be loaded
    • Handle complex primary and foreign keys
  • Apply distinct row selection and where-clause to any source data mapping
    • Minimize duplicate hashes and leave out rows you don’t want to load
    • Especially useful during development
• Partial solution generation if parts of model fail validation
  • You can restart the Data Vault generation process even though some classes have failed verification. The failed classes will internally be marked as non-implemented before starting the process.
  • Deploy parts of model even though other parts are broken.
• Semi-Automating SameAs tables
  • Now generated: SameAs-orchestration procedure calling stub procedures for populating the sameas tables. Complete these stub procedures manually, if applicable (many sameas tables can’t be automatically populated using any logic due to the nature of the…sameasness)
• New datatype: geometry (to be used for Business objects only)
• Database filter and Environment type added to Database server metadata in Visual Paradigm
  • Create separate server objects for dev, test and prod environments as well as for actual source systems.
  • For visual comparison of the different environments
• Telerik UI component upgrade

Fixes

• Reduced left joining when hashing
• Column sizes now created correctly when materializing views as tables
• Materialization procedure now correctly renamed if moved to the raw vault schema
• RAW_ prefix not added to tables when importing table metadata, if the table already has the RAW_ or OUT_ prefix in its name

There are changes in Visual Paradigm Stereotypes (SEStereotypes.xml), so you will need to re-import the file.

If you are upgrading from an earlier version, check the Help.Upgrades menu, and run each of the Upgrade commands that have a version number that is bigger than the one you are upgrading from. These commands analyse the currently active model for issues that you may need to look at.

Release date: 2022-09-30

Features

• Database connection
  • Connect to source and target servers using native drivers when connecting to Microsoft SQL Server variants, and ODBC otherwise
  • In Visual Paradigm, model the Database servers in Infrastructure
    • Use Database server stereotype
    • Add a working Connection string
    • Use placeholders %USERNAME% and %PASSWORD% as necessary. These can be replaced by actual values in the D♯ Engine UI before opening the connection
    • For the server that contains the generated DW solution, set Default SQL Target to True to enable class based browsing.
• Explicit Materialization now available for all class types
  • Yes = Materialize
  • No = Don’t materialize
  • Automatic = Materialize if class’ Volume is High or more, or if the class is part of a transformation (for example, directly used as source in a mapping to a BO class, or as part of a transformation path in such a mapping).
• Key Groups now fully functional
  • For each class, group sources together that use the same Business key to mean the same thing
  • D♯ Engine will then apply the specified prefix in the hashing function, integrating the data within the same key group, and avoiding hash collisions from other sources that may use the same Business key to mean a different thing, or other variations of incompatibility.

User Interface Components

• New: Class Implementation
  • Alphabetic list of classes, grouped to Implemented and Non-implemented classes
    • For Raw Model classes, shows
      • Staging Area objects (tables)
      • Hashing objects (work tables and corresponding hashing procedure)
      • Raw Vault objects (tables and corresponding load procedures)
      • Business Vault objects (class view/table with corresponding materialization procedure)
    • For Business Model classes, shows
      • Business Vault objects (generated tables as well as class view/table with corresponding materialization procedure)
  • Primary use is to quickly find a class and its implementation. Use together with the SQL pane to verify the implementation.
• New: Servers
  • Tree view showing the contents of any registered database server
    • Shows Databases, schemas, tables, views, columns, primary keys and foreign keys
  • Primary use is to browse structure and data and to provide mapping information
  • Default SQL Target is used for providing example data to HTML documentation
• Improved: Data Grid
  • Now has the same syntax-aware SQL editor as the SQL pane
  • Click anywhere on any class or data object, real or virtual, to browse its data
    • For classes and generated tables and views, queries the data from the SQL Target server, which can be set in the Properties of the database server
    • For real server objects, shows the content of the actual clicked table or view
• Mapping Editor
  • Brings the Source-data-to-Model-element mapping process inside D♯ Engine
  • Use the familiar spreadsheet layout to edit the mappings
  • Create new mapping files with automatic numbering
  • Create mappings by dragging and dropping from the Server and Conceptual model panes, either row by row or in larger batches
    • For instantaneously correct pairing, click on the model elements in the order you want them to be inserted into the file
  • Save & refresh!

Fixes

• # as attribute visibility (protected) in Visual paradigm now correctly generates the hash function call for the attribute value in the Business Layer
• Fixed an issue that caused some UI Panes to remain without a name/caption
• Model analysis when opening a model now identifies classes within the same domain (Business Model, Raw model) that have the same name
• Checking the AlreadyHashed-column in a mapping file now correctly handles the column when hashing

There are changes in Visual Paradigm Stereotypes (SEStereotypes.xml), so you will need to re-import the file.

If you are upgrading from an earlier version, check the Help.Upgrades menu, and run each of the Upgrade commands that have a version number that is bigger than the one you are upgrading from. These commands analyse the currently active model for issues that you may need to look at.

Release date: 2022-09-06

This is a minor maintenance release.

Feature

• In the Visual Paradigm Data Vault 2.0 element, Added parameter Nullhash pattern, which is a string value used to generate the hash value corresponding to null, by repeating the string until it fits snugly in the generated hash columns. Default value “-“.

Fixes in the Visual Paradigm Data Vault 2.0 element

• Trimmed list of available Hash functions to MD5, SHA 256 and SHA 512. The selection sets the hash column sizes and the hashing function used in the database
• Installation package now correctly generates the CalculateHash procedure during Full install / DW install, depending on the hash method selection

If you are upgrading from an earlier version, check the Help.Upgrades menu, and run each of the Upgrade commands that have a version number that is bigger than the one you are upgrading from. These commands analyse the currently active model for issues that you may need to look at.

You will need to re-import the SEStereotypes file into Visual Paradigm for this release.

Release date: 2022-08-19

This is a maintenance release focused on bug fixes.

Fixes and tweaks

• The text %DATABASE% is no longer present in the deployment package
• Deployment does not back up staging area views anymore if their structure should be changed
• Bracketed table names not used anymore in table name comparisons when running the Full DW Installation command
• Corrected the bug which caused the wrong datatype to be assigned to some view columns
• All database references removed from Azure deployment package
• The ZeroToNull transformation fixed so that it does not assume the input data to be a string constant
• Fixed an error which prevented the SQL comparison operator “different than” (<>) to be used

If you are upgrading from an earlier version, check the Help.Upgrades menu, and run each of the Upgrade commands that have a version number that is bigger than the one you are upgrading from. These commands analyse the currently active model for issues that you may need to look at.

Also see the Readme.txt file in the root directory of the installation for information when to re-import the SEStereotypes file.

Release date: 2022-06-30

Features

• Support for automated model based Business Model implementation
  • Major functionality boost to Mappings: describe transformations and formulas using model elements and paths, or any target platform compatible code.

Fixes

• Fixed a crash that happened when building a deployment package if the SQL pane was open

Before installing any new generated code with this version, please run the command Help.Upgrades.Upgrade to 3.1.0 for each model to be implemented. This will analyze the model and let you know which issues you need to look at.

In general, if you are upgrading from an earlier version, check the Help.Upgrades menu, and run each of the Upgrade commands that have a version number that is bigger than the one you are upgrading from.

There are changes in Visual Paradigm Stereotypes (SEStereotypes.xml), so you will need to re-import the file.

Release date: 2022-05-13

General

• Branding change: No more SmartEngine, welcome D♯ Engine
  • Executable name changed to DSharpEngine.exe
  • “Smege”, among friends, does not make sense anymore
  • Suggested nicknames: Dshege, Dege, Ege, Endshini, Dshini, …
• License agreement must be accepted in order to run software
• Requires new license file, as the UID changes due to the software name being part of the UID definition.
• Monthly support licenses available: license is target but not developer specific, and can be easily distributed within the company for support purposes (in accordance to support agreement).

Features

• Support for adding new target platforms
  • First new platform PostgreSQL: just select PostgreSQL as target in the Infra section’s Data Vault 2.0 settings
• Per project, possibility to turn automatic index management on or off.  Useful for Azure projects, as Azure continually optimizes indexes based on actual use. Turn this property on or off in the project’s Data Vault 2.0 settings. 
• UI library upgrade
• Deployment mechanism changes
  • for quicker installations, work and error tables are now simply dropped and recreated (if needed, manually backup any error tables containing unhandled error cases before deploying)
  • all table and view level changes and especially failures while deploying are logged in high detail (remember to always check the log after deployment!)
  • more advanced column datatype change handling, for example
    • if a column’s new datatype is incompatible with the existing column, the column will be backed up and a new column will be created with the correct datatype. Populating the new column should then be done manually.
• Datatype changes
  • new datatypes for specific number of decimal places: decimal1, decimal2, …, decimal38. The default decimal type still supports four decimals. 
  • idstring is deprecated. Use code instead. 
  • long and short are deprecated. Use bigint and smallint instead, respectively.
• SQL Pane
  • View SQL code for selected database objects

Fixes

• Overflowing Element names are now forced to their platform-dependent maximum length by removing less important characters (vowels, underscores, …) until the correct length has been reached.
• Unnecessary type casts removed from generated views, making them slightly faster

Before installing any new generated code with this version, please run the command Help.Upgrades.Upgrade to 3.0.0 for each model to be implemented. This will analyze the model and let you know which issues you need to look at.

If you are upgrading from a version earlier than 2.7.0, also run the Help.Upgrades.Upgrade to 2.7.0 command.

There are changes in Visual Paradigm Stereotypes (SEStereotypes.xml), so you will need to re-import the file.

Release date: 2021-12-31

• Support for automatic source name inclusion in hash
• • Even further separate data by source system by automatically adding a source system specific component to the hash, or
  • Avoid hash collisions between different source systems that are using same key for different real world phenomena
  • No longer need to create an OUT_ only for this purpose
• Same-as support
  • Can be used for
    • Loading versioned data into a class but displaying only the newest version
    • Resolve key changes when old and new keys are known (for example, person id changes)
    • Together with source name hash inclusion, further separating the data in the raw vault by source system on hash level, but consolidating them in the view layer.
  • Generates same-as table
  • The class view may be configured to
    • Redirect all incoming references to a master row (ie from old person id to new person id, or newest version)
    • Redirect and also filter out irrelevant versions from the view (don’t show old person id’s at all)
• New transformation: add !Hash to your mapping file’s Transformation column and the field will be hashed
• New command: analyze max length of data for any column

Fixes

• Element names are now trimmed when reading from Visual Paradigm
• Classes with Implement=false are not reported on application startup as having unmapped properties
• If Duplicate handling for a class has been defined as Force uniqueness, the unique index is no longer included in the calculation of the reference hashes, now producing the correct hashes
• Commands/menu structures have been reorganized

Before installing any new generated code with this version, please run the command Help.Upgrades.Upgrade to 2.7.0 for each model to be implemented. This will analyze the model and let you know which issues you need to look at.

Also import the included SEStereotypes.xml as described here. Changes:

• Key consolidation and Key groups parameters added to the following Stereotypes:
  • Party, Place, Thing
  • Moment – Interval
  • Role
  • Description
• See Metadata for description of how to use the new parameters

Release date: 2021-09-30

• Integration with the newly released D♯ Developer Community pages
• All tutorial material moved to the community site
• Unneeded indexes no longer created for materialized views’ reference hash columns
• SmartEngine now shows a descriptive error message if the mapping file is locked when opening a model
• The database name is no longer converted to an object name. Eliminates the need for manual fixing of database names containing the ‘-‘ character when generating SQL code.
• Behind-the-scenes cleanup of script code

If you are using materialized views, you may want to re-install them using this version. The reinstallation drops the unnecessary indexes, making the materialization procedure more efficient.

Release date: 2021-06-23

• Major Telerik UI component upgrade and behind-the-scenes UI work, 3 years worth of updates in one go
• Descriptive error message when deployment fails because of unmapped properties
• Unmapped properties listed in separate view after project opened
• UI commands for installing the ghost row, dropping tables and querying views

Release date: 2021-05-14

• More efficient view layer, eliminating real-time hash calculations when referencing Description classes, by
  • fetching the hash from the appropriate hub over a join, and
  • indexing the BK column in the referenced hub
• Deployment packages can now be built for individual database objects directly from the Server Model pane
• Long index name now shortened if the name exceeds 100 characters

When replacing and older K_ view (or materialization procedure) for an efficiency boost, also remember to re-install all Description type objects that this view references as *_hash columns, so that the indexes are created for the referenced xhub’s key attribute column. This index is required for the new view to be efficient. As an alternative to installing the entire Desctiption classes, you can also make a deployment package with just the individual xhub tables.

Release date: 2021-03-31

• License handling
  • A license now has a target, typically a specific customer or a specific solution
  • The target name is visible in all generated procedure and view headers
  • The target name is visible in the SmartEngine main window’s title bar
  • SmartEngine displays a warning in the main window’s status bar once there are less than 30 days left before the license expires
• New analysis functionality
  • Link-to-link –situations are automatically detected when analysing the Conceptual model
  • UI command: Unmapped attributes listed (attributes with no corresponding data source)
• Datatypes
  • Property pane shows the base data of the datatype, including description and how it maps to its appropriate SQL datatype
  • Fix: varchar datatypes now implemented with correct template-based sizes
  • If changed, the deployment package will try to correct the column sizes for existing tables included in the deployment package
• Comments-column added to most table types
  • Existing tables will automatically be altered if included in the incremental deployment packages
• HTML documentation
  • Data Lineage information previously only available through the UI now added to the HTML ER-model documentation
  • For each table, diagrams with incoming and outgoing data flows on table level, and
  • Column level in- and outgoing data flows
• UI reports progress when generating new objects
• Fix: Large deployments (millions of rows of SQL code) no longer halt the SQL code generation due of OutOfMemory errors
• Classes with the Implement parameter set to false can now be “rough and incomplete” without affecting the DW automation
• The calendar dimension now correctly installs when using the Full DW Installation command
• Automatic horizontal scroll bars added to text windows for added convenience

Release date: 2021-01-19

• URL datatype length increased
• Column sizes harmonized
• Now catches non-unique column names when hashing
• When deploying, existing staging area table columns will not be altered

Release date: 2021-01-12

• Commands
  • List potentially unused database objects
  • Clear DW tables
  • Analyze conversion- and reference errors
• “Complete incomplete” data
  • Rule based mechanism for marking missing / dependent data as deleted when loading incremental data (but complete within that data set)
• Error analysis
  • Conversion / reference error logging
  • Added error tables to installation
• Business Layer
  • Soft Delete now parameterized
  • Satellite naming fixed
  • Custom SQL mapping implementation
  • Updated Soft Delete tutorial

Release date: 2020-12-04

• Modeling
  • Possibility to have an identically named Business- and Raw Model classes. Business Model class will “override” the Raw Model class, which will relocate to the rav vault schema.
  • Separate Sterotypes to denote Business Object Archetypes
• Naming of Business Layer objects (formerly “K_” views) now parameterized
• Tutorials finalized
• Fixes
  • Fixed bug in class-to-class mapping
  • Now produces an error message if a primary/business key component does not exist in the class
  • No recursive hash procedure calls when hashing a self-referencing class

Release date: 2020-10-13

• Application
  • File based license check added
  • Rebranding: “K#” replaced with “D♯” everywhere
  • Opening export file (project.xml) directly instead of wrapping it in a D♯ project file
  • Model based solution for the equivalent of multiactive satellites
• Mechanism for handling hashing errors (parameter based “remove duplicates” before loading)
• Created Tutorials for self-learning and training
• Fixes
  • Association can now be drawn both ways
  • Full DW install now always installs the entire model, not just the extended selection
  • Diagram commands streamlined
  • Removed all unused scripts

Release date: 2020-09-03

• Installation package bug fix
• Added several demo scripts with corresponding demo data
• Re-introduced the Data Grid for demo and data debugging purposes

Release date: 2020-06-15

• General cleaning up
• Added default value to nullable columns

Release date: 2020-06-11

• Support for Ellie as modeling tool
• UI
  • Property pane for displaying selected object’s properties
  • Changed to a longer delay time for text searches in the tree views: the component waits three seconds before the search starts running after you have entered something in the search box. Pressing return lanches the search immediately.
  • Command execution times available in the status bar.
  • New incremental deployment package available.
  • Raw Vault and Data Flow diagrams now available.
• ETL parameters
  • Soft delete possible if data is no longer present in any source it has been loaded from previously.
  • Turn duplicate hash handling on or off.
• Commands
  • Hash, Load as well as Hash & Load a class directly from the UI.
  • Analysis to find datatype conversion errors.
  • Analysis to find differences between model and database (tables and views).
• Business Layer
  • Classes with Volume = High or more are automatically materialized using a table and a populating procedure instead of a view.
  • Better indexes generated for K_ view usage.
• Legacy
  • v 1.4 scripts removed, resulting in a cleaner menu structure. All visible commands actually do something.

Release date: 2020-01-27

• Completely re-written internal architecture. New object model to support Data Vault 2.0.
• Internal Object Model rewritten without using interfaces. More flexible architecture.
• Full support for standard DV2 objects: hub, satellite, link (many variations: standard link, link-as-class, non-historized link), state satellite (effectivity satellite), PIT. Business vault objects generated but not automatically populated. Views generated on top of both Raw Vault and Business Vault tables.
• Model-based mappings (simple mapping as well as pivoting) automatically generate the Business Layer views.