DevExpress tDx to tCx Automatic migration - Felix John COLIBRI.

• abstract : automatic migration of DevExpress tDx grids to the Unicode tCx grid, using a tool which splits the forms into single grid form in order to build, fine tune and test the .DFM / .PAS transformation tool.
  
• key words : DevExpress - tDxDbGrid - tCxGrid - Unicode - .PAS parser - .DFM parser - automatic transformation - form extraction - unit test
  
• software used : Windows XP Home, Delphi 6
  
• hardware used : Pentium 2.800Mhz, 512 M memory, 140 G hard disc
  
• scope : Source projects : Delphi 5 to 2007 - Target projects : post Delphi 2009 unicode projects
  
• level : Delphi developer
  
• plan :
  
  • Devx tDx to tCx migration
    
  • tDx to tCx migration strategy
    
  • Splitting the tForm
    
  • Removing Form Inheritance
    
  • Creating the mock test tables
    
  • Creating the test projects
    
  • Migrating to tCx
    
  • Workload
    

1 - Devx tDx to tCx migration

This occurs mainly when the customer shifts to post 2009 Delphi versions, where Unicode is required. The tDx component suite has not been ported by DevExpress to Unicode. Instead they created a new grid and component set, the tCxGrid and other tCx components, which are unicode enabled and are maintained with the successive versions of Delphi.

There are several ways to tackle this problem

• migrate the tDx set to Unicode. We have seen this with one customer. First of all this is a massive job. Moreover, the migrated tDx components will no longer evolve (new components, new Windows evolutions etc)
  
• switch to another Grid (TMS comes to the mind). The effort is even greater than migrating tDx to tCx
  
• build a custom grid starting from some basic Unicode grid (tDbGrid) and add feature to mimick the tDxDbGrid. This is possible for very simple uses of the tDxDbGrid, but quickly becomes unmanageable when the customer starts to use more sophisticated features of the grid. Basically you are trying to duplicate the many years of developpment of the DevExpress team. This is a hopeless job
  

2 - tDx to tCx migration strategy

3 - Splitting the tForm

3.1 - Isolate the tDx containing forms

This avoids parsing of the form and units not concerned by the migration.

This step involves the creation of a separate \tForm for each \tDxDbGrid.

3.2 - Algorithm

• for each tForm containing some tDxDbGrid, we created one or several simpler forms containing the minimum content
  • one tDxDbGrid
    
  • the associated tDataSet and persistent tFields
    
• those forms were flattened : we created one form only for the leaf forms and removed all the inheritance functionality. This allows to open the form without the inheritance hassle ("cannot find the ancestor" etc)
  
• each grid was associated with a .DPR
  

3.3 - Splitting the multi-grid forms

4 - Removing Form Inheritance

• a base form with some panels and PageControls
  
• a middle descendent form with a tDxDbgrid, a tDataSource referenced by the grid and an empty tDataSet referenced by the tDataSource
  
• a leaf descendent with
  
  • a filled tDataSet (with SQL statements and persistent fields
    
  • a descendent of the tDxDbGrid with more properties, with DxDbColumn components and events
    
  Note that if the leaf node does not modify the tDataSource (no additional OnDataChange, ReadOnly etc), this component is not present in the .DFM of this form
  

The technique is then to

• start with the last descendent(s)
  
• gather the components properties and values at this level
  
• analyze the ancestors using the inheritance chain, and
  
  • add to the leaf form the ancestor's properties and values not yet present
    
  • add all interesting components not present at the leaf level (tDxDbGrid, tDataSource, tQuery)
    

Here is a sketch of this process:

and a partial example:

This is what we call "flatening the forms"

• remove the containers
  
• remove the unnecessary components
  
• include any \tDataSource, \DataSet present in the ancestor but not in the last descendent
  
• consolidate the \tDxDbGrid, \tDataSource, \DataSet properties
  

5 - Creating the mock tables

5.1 - The data used by the grids

The grid is totally unaware of the origin of the data: it is just fed with rows and column values from some tDataSet.

It is therefore quite easy to replace the original tDataSet with an artificial tDataSet, provided its column types are matched to those expected by the grid column components.

5.2 - Grid Columns

Here is an example with a simple tDxDbGrid operating on the "Cars" Devx demo dataset:

In this simple example

• tDxDbGrid1 has a DataSource property pointing to DataSource1
  
• DataSource1 has a Dataset property pointing to Table1
  

There is no field type or field size information. And this is enough to present the data in the grid, at design time and runtime.

This is not enough for building a suitable mock table. At the end of the day, the grid only displays strings. However buiding a mock with only string could violate some statements in the tDxDbGrid events, which could use Integer, Float, Boolean or DateTime expressions.

Could we use a general SQL statement to find the FieldType and Size information ?

Certainly, we could just duplicate the Delphi IDE technique by loading Table MetaData.

This could however become quite involved if the actual query is the result of complex JOINS or stored procedure.

5.3 - Using Persistent Fields

We added the persistent fields to our Cars application, and here is the .DFM

When persistent fields are present, we use this simple algorithm to build our mocks :

• for each tDxDbGrid we locate the DataSource
  
• this tDataSource is associated with a tDataSet
  
• the tDataSet contains the tFields, which contain the field name and the field type and size
  

5.4 - Using Data Generators

Using the data type and size, this is quite easy

• numeric, boolean and date can be easily randomly generated
  
• for strings, we can use
  
  • random string, but this is not very appealing
    
  • the classical "lorem ipsum" used in publishing, graphic design and many data processing random text generators
    
  • the first pages some book like "Tom Sawyer", much more entertaining then this latin text
    

In fact, we long ago extracted values from the standard database demos (MastApp, FishFact, NorthWind, Pet Store...). So we have lists of realistic looking "Cities", "First Names", "Street", "Companies" which could be used.

To allocate a specific data generator to a field, we simply listed the Forms / Tables / Fields and enrich this list with generators

• for standard types (Date, Currency) the generator can be automatically generated
  
• for string fields we manually associate fields with generators.
  

-field_name----- field_type--- size ----allocated_generator----                                            name min max order_update.dfm detail_query id_customer TStringField 5 integer 100 199 id_order TStringField 7 integer_string 1000 1999 order_date TDateTimeField - date 2015 2016 order_amount TCurrencyField - currency 500 3000 sales_tax TCurrencyField - currency 100 200 code TStringField 3 update_clerk TStringField 5 update_date TStringField - date_string 2015 2016 update_slip TStringField 30 string_5 20

We only added the min / max values of the Integer, Date and currency fields.

In this customer / sales table we specified :

• the min and max of the Integer fields
  
• "company"
  
• "address_street"
  
• "us states", with a maximum of 5
  

 
    -field_name-----  field_type---  size  ----allocated_generator---- 
                                           name            min  max 
customer_order_address.dfm 
  master_query 
    id_customer       TStringField      5  integer         100   199 
    id_order          TIntegerField     -  integer         1000  1990 
    customer_name     TStringField     30  company 
    street            TStringField     30  address_street 
    state             TStringField      5  us_states       5 
    city              TStringField     24  city 

Once we have allocated generators to one table, we look for fields with the same name, type and size, and allocate the same generators to those fields. Here is the result of another table having "company", "city" etc:

-field_name----- field_type--- size ----allocated_generator----                                            name min max customer_contract.dfm customer_contract id_customer TStringField 5 integer 100 199 company TStringField 30 company street TStringField 30 address_street city TStringField 30 city contract_number TStringField 15 start_date TDateTimeField - date 2015 2016 end_date TDateTimeField - date 2015 2016 payment_terms TStringField 3 company_category TStringField 3

5.5 - More complex cases

• if the tDataSet is placed in tDataModule. We simply analyze this tDataModule to import the tDataSet in our simpler form
  
• the persistent fields have not been created. We then can create those fields on a copy of the original Form
  
• the tDataSet is assigned by code, not in the .DFM. In this case, we look at all the tDataSets and use one them which have at least all the fields that the grid has
  
• the query is created and linked to the grid by code. We can generate the field information at runtime, reading the tFields or the tFielDefs created when the table is opened. However we have to locate the form, and therefore explore, dig and spelunk. Just the opposite of what we tried to achieve.
  
• if the grid involves lookups, more datasets might be brought in, with integrity checks between the tables
  
• if all fails, we can always assume tStringFields, and correct the type if the grid events if the form does not compile (for instance an event uses AsIntrger or AsDateTime for some field)
  

5.6 - Using the mocks

Here is the situation after the split / flatten / mock operation :

and the "split / flatten / mock" steps can be described by :

// -- split ForEach tDxDbGrid in the leaf form     create a tForm and a .DPR     copy the tDxDbGrid     // -- flatten     copy the ancestor's properties     // -- mock     find the tDataSource and the tDataSet     using the tDxDbGrid's tDxDbColumns and tDataSet's tFields         create a Table         fill the Table

6 - Creating the test projects

6.1 - One .DPR per Form and Grid

6.2 - Compiling the projects

• the tDxDbGrid events code uses other components which were not included in our simpler form, like
  • tables for master detail relationship or lookups
    
  • calls to other forms, like dialogs
    

An intermediate solution is to include some of the external items and calls in the simpler form.

There obviously is a tradeoff to the number of external items which could be included.

And since the form has an associated .DPR, we can run the project and examine the effects of user interaction and of the grid events.

6.3 - Batch compilation

Failures can saved in a log.

7 - Migrating to tCx

Now is the time to migrate to tCxGrid :

This migration was performed by adapting our migration tool to the tDxDbGrid. As already presented in several papers, this .PAS / .DFM conversion tool :

• starts with a list of types to be migrated (Bde types, tDx type etc)

  TdxBlobEdit TdxButtonEdit TdxCalcDisplay TdxCalcEdit TdxCalculator TdxCheckEdit TdxCheckEditStyleController TdxCurrencyEdit ...

• the list of properties, members an events of each type are constructed using RTTI :
  • for the new post Delphi 2010 applications, the "new" RTTI
    
  • the "old" Delphi RTTI for pre 2010 applications
    
  In our case, the tDx components are analyzed using the "old RTTI". In this case only the Public and Published properties and events are computed, but not the methods

  Here is an extract of the tDxDbGrid

  // p_rtti_extract_pme.exe - D6 version - NO methods (public) // TCustomdxDBGrid TdxDBGrid   properties     Align     Anchors     ArrowsColor     AutoExpandOnSearch     AutoSearchColor     AutoSearchTextColor     BandColor     BandFont     BandMaxRowCount     BandRowCount     ...   events     OnAddGroupColumn     OnBackgroundDrawEvent     OnBandButtonClick     OnBeforeCalcSummary     OnBeginDragNode     ...

• all the customer forms with some of the target types are analyzed:
  
  • first we find the names of globals, locals, parameters and members with the target types
    
  • we then find all the types, properties, members and events actually used in the application at hand.

    Here is a small extract

    tdxdbgrid    Align      alClient         alTop    Anchors         [akLeft, akTop, akRight, akBottom]    ApplyBestFit         ApplyBestFit(dbgMaitre.Columns[i]);    AutoExpandOnSearch         False    BandColor         13224393    BandFont.Charset         ANSI_CHARSET         DEFAULT_CHARSET    BandFont.Color         clBlack         clBlue         clNavy         clRed         clWindowText

    This step usually drastically reduces the number of items to handle. In our case :

    • only 33 types among the 52 tDx
      
    • for the tDxDbGrid, for instance, only 180 among the 600 + available
      
      
  • we define a replacement strategy for each of the remaining items found in the forms. This can be
    • either a simple name replacement ("DatabaseName" is replaced by "ConnectionName")
      
    • or a more elaborate computation using code
      
    Both the .PAS and the .DFM are modified
    

• visual examination
  
• behaviour tests
  

Then some behavior must be checked: does the tCx sort as the tDx, is the filter working in the same way, are the subtotals grouped in the correct order etc.

We already explained, that to check a transformation, compilation is not enough: you must execute the project and actually load the form to catch some .DFM bugs which passed the compilation but will fail when the .DFM resource is loaded. Our single form project is well suited for this test. And in addition logs of the compilation and the execution error are created.

7.1 - decreasing effort

We prefer a more incremental approach, since all conversions are not fully understood.

The first couple of tDxDbGrid migrations are the more difficult. We have to find the replacements of each property and value of those first forms and include them in the migration tool. Mapping tables have to be designed and populated, ccde transformation designed and written etc.

But as time goes by, the new grids only contain incremental new properties and values. So the time spend to transform the additional forms quickly decreases.

In fact we use a follow up file to check what has been migrated and what has not yet been covered :

• we start with
  
  • the list of all forms to be migrated: the "form todo list"
    
  • the list of all properties to be migrated: the "property todo list"
    
• repeatedly, we
  
  • select a form
    
  • check if the form contains new properties (not contained in the "property todo list". If some are found:
    • transform them
      
    • remove them from the "properties todo list"
      
  • remove the form from the "form todo list"
    

This is in strong contrast to the manual migration route where, even if you perfectly know what mapping to apply, you still have to manually make the changes for every single form.

7.2 - Regression test

This is where the batch compilation and execution with logs come in handy.

8 - Migrating the original project

So we use a two phase approach :

• split / flatten / and mock to build and fine tune the transformation tool
  
• apply the transformation tool to the whole application
  

9 - Workload

9.1 - The basic migration tool

How much did this tool costs ?

• it uses the scanner and parser of Niklaus WIRTH's "Algorithm + Data Structures = programming" book as well as snippets from the Zurich P4 compiler. The current version is based on a Delphi Grammar and a parser generator. Several versions of this recursive descent parser have been published and can be downloade from our sites
  
• for the .DFM part, we published an article about the .DFM grammar along with a .DFM parser. The parser used here is an upgraded and up to date version of the published one. The parser generates a .DFM tree (similar to DOM) which is easy to visit and modify
  
• the RTTI analyzer was coded over time, with the "Old RTTI" as well as the "New RTTI" versions
  
• creation the type -> variable -> property / method / events -> transform tool was constructed for our first migration job (migrate a 2.600 unit project from BDE to Ado)
  
• the data generators were also coded a couple of years ago
  

9.2 - The split / flatten / mock

9.3 - The adaptation to Devx

This heavily depended on the complexity of the grids used by our customer.

9.4 - Other migration techniques

9.5 - Cost of future DevExpress migrations

10 - Remarks

• the customer software was particularlly well suited for our automatic approach : many rather medium complexity grids.

  In other case, the project revolves around 2 or 3 central grids, using a large portion of the tDxDbGrid functionalities.

  If the conversion tool does not handle them all, the effort to include them is more important and the cost cannot be averaged over several grids.

  Now if the customer has several of those sophisticated grids, then the benefit becomes again obvious. In any case, the gained knowlege can be used for other migrations.

• our tool is more .DFM oriented than .PAS, in contrast to Database migrations, where data access components are often spread all over in the .PAS. For grids, most of the action lives in the properties. In fact the Grid editor's business is to encapsulate slick grid behavior in properties : the developer selects a couple of properties instead of writing sorting, filtering, pivoting code

• of course we included in the same migration tool the other tDx components (tDxEdit, tDxDbGridButtonColumn, tDxDbGridLayoutList etc)

• the individual projects containing the simpler forms are delivered to the customer, should he want to use them for some checks. However, those projects are not the final acceptance of the migration. The user still has to submit the result to his QA team and run acceptance tests

• did we build a universal DevExpress migration tool ? Certainly not. We already told that we took a minimalist approach, to be able to convert the units at hand. For new customers, we only will have to add other components, properties, methods, events.

  To build such a exhaustive product, we would have to cover all tDx types, properties, methods and events. Code the transforms and test them.
  

10.1 - Unit Test

There are differences :

• unit test is centered around code (although some GUI test can be conducted). Our tool concentrates on visual presentation and properties
  
• unit test usually works on the statement / procedure level, whereas we have a broader granularity
  

10.2 - Cost / benefit evaluation

• the high upfront cost of building the tool. In our case, most of the techniques were available and the migration tool already created for other domains.
  
• can slow down the migration of the first forms, since we have to build the tool (or adjust to the customer's tDx habits)
  
• still requires the creation of individual tDx to tCx property, method and event conversion rules. But it makes the process easier to test.
  

• can avoid to loose or miss some properties (wrong .TXT find and replace of the .PAS and .DFM, for instance)
  
• can report on the tDx components or properties without any tCx direct equivalent. They can be reported on a per form / grid basis
  
• can be run many times, allowing improvements as the tests sends feedback. With manual conversion the individual developer gains insights in the process, but this know how cannot be easily transfered to other developers / companies
  
• the creation of separate projects with a single tForm and a single tDxDbGrid allows better and quicker tests
  
• is able to perform regression tests, with compilation and execution error logs
  
• will speed up the customer feedback with earlier deliveries. It is even possible to perform the migration before any Data Access / Unicode / 64 migration (since tDx and tCx can be installed on Delphi 5 and higher).
  
• can be used for early "proof of concept" jobs
  
• the next customer only pays for incremental costs (adjustment to his tDx choices)
  
• is more efficient for whole multi-grid project migration. This, of course, is difficult to proove
  

10.3 - Other uses of the .PAS / .DFM transforms

• quite similar to grid switch is report switch. You might have to migrate from Quick Report to Fast Report or from Crystal to Report Builder. In this case too, the properties rather than the code are central to the conversion effort.

  There is an additional complexity with the page change rules (sub-totals should not appear at the top of the next page etc)

  Reports also involve not only a generation change but also a Vendor change, and therefore often an architecture change

• migrate from one Sql Engine / Data Acess component set to another Engine / Data Access set.

  The split phase is not exactly the same. Instead of using the
    :tDxDbGrid -> tDataSource -> tDataSet => simpler_form
  pathes, we use
    tDataset -> simpler_form

  In this case we often have to use simpler forms with a simple main tDataSet, and several of its associates tDataSet (master / detail relations). The focus here is on the data access, which involves

  • more code handling
    
  • tables and their dependency
    
  For legacy code, the split and mock approach finally offers the testing tool that has eluded us for so long (afraid of navigation, database knowledge ...)

• other candidates could be the replacement of compression tools (.ZIP) or Tcp/Ip suites. We would however carefully evaluate the cost / benefit approach before using our tool on those more simple component changes.

• separate the old "Dataset / computation / display" forms into separate "Model / View / Control" tDataModule, tForm and Unit

  Splitting the tForm into a tDataModule and a Unit should not be too difficult, but transforming do-it-all events is a little bit more challenging :

• the same technique can be applied to extract the Model and Controller and replace the View with Web Forms or Web Services communicating with any GUI layer, like PHP and JavaScript

• a little bit further along the road, introducing Business Objects, and then replacing Queries with some Persistence Layer.

• finally the extraction could be used by software archeology (wikipedia : "the study of poorly documented or undocumented legacy software implementations, as part of software maintenance") to help the refactoring the code, and especially to introduce some unit tests

  Or extracting some deeply nested buggy piece of code to to isolate it and fix it using the mocks

  Granted, transforming huge legacy code looks a little bit far fetched, but the tool can be used as an aid for the transformation.
  

11 - References

• DevExpress site which contains freely downloadable documentation in .PDF and Help formats
  
• documentation and samples of the previous tDx products is no longer available. We would be happy to migrate their demos, which, a usual, use the most advanced and less intuitive properties that the Vendor found interesting to show.
  

• other Delphi migration articles :
  
  • Database access component migration :
    
    • BDE to Ado migration
      
    • Bde to Unidac migration
      
  • Unicode migration (in French)
    
• articles about .PAS / .DFM scanners and parsers
  
  • the Delphi 5 grammar we used to generate the Delphi parser
    
  • Expression Interpreter : a small expression grammar with scanner, parser and AST (Abstract Syntax Tree) interpreter (in French)
    
  • .DFM Parser : grammar and parser of .DFM files. This is an early version of our current parser
    
  • Sql Parser : a grammar and parser for Interbase / Firebird Sql requests (SELECT, INSERT and UPDATE)
    
• Equivalence table between tDx and tCx components : the table we use for the migration of the non grid tDx components
  

12 - Your comments

• please tell us at fcolibri@felix-colibri.com if you found some errors, mistakes, bugs, broken links or had some problem downloading the file. Resulting corrections will be helpful for other readers
  
• we welcome any comment, criticism, enhancement, other sources or reference suggestion. Just send an e-mail to fcolibri@felix-colibri.com.
  
• or more simply, enter your (anonymous or with your e-mail if you want an answer) comments below and clic the "send" button

  Name :
  E-mail :
  Comments * :

• and if you liked this article, talk about this site to your fellow developpers, add a link to your links page ou mention our articles in your blog or newsgroup posts when relevant. That's the way we operate: the more traffic and Google references we get, the more articles we will write.
  

13 - The author