One common piece of functionality in many business applications is the ability to import data from an external source. This could be anything from invoices and orders to inventory forecasts and patient medical histories. Quite often if this type of process is not considered in a generic context you end up with a solution which is too specific and is only suited for the immediate need. As time goes on you can end up ‘reinventing the wheel’ many times over leading to a maintenance nightmare.<\/p>\n
If you are lucky enough to be dealing with clients using some sort of standard (EDI, HL7, etc.) or if you are able to establish the ‘standard’ file layout you have passed a major hurdle. At this point you can develop the process of getting the ‘messages’ (raw data records) into the database. In general, my vote for this phase is a process consisting of stored procedures which are triggered by some type of scheduled job. Remember, at this phase you are only importing the ‘raw’ data into a preliminary set of ‘raw data tables’; no business rule validation is taking place.<\/p>\n
From an auditing standpoint I tend to keep copies of the data at each stage of the process. By this I mean I copy the original file to an archive location, import the file into a raw data table, process the raw data into another table where datatype checking and message ‘completeness’ (proper length, correct number of segments, etc.) is performed, and then bring the records into the data tables of the application itself (and apply business logic). This way if there is any question on what happened to any given transaction, I can trace its path step by step.<\/p>\n
General Process Overview<\/strong><\/p>\n One way in Powerbuilder to bring the data into the application is through the use of the LibraryExport method. With this technique you can set up a .pbl file which contains the datawindow object for a specific dataset you are expecting from the records in the transfer table. A nice feature of this is if there is some minor change to the data being sent (additional column, change in column, etc.), you can easily and quickly make the change, deploy the .pbl file, and you are back in business without having to rebuild the entire application. This assumes the change doesn’t require some processing logic change within the application.<\/p>\n One common piece of functionality in many business applications is the ability to import data from an external source. This could be anything from invoices and orders to inventory forecasts and patient medical histories. Quite often if this type of process is not considered in a generic context you end up with a solution which…<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[3,10],"tags":[29,12,16],"_links":{"self":[{"href":"https:\/\/anvil-of-time.com\/wp-json\/wp\/v2\/posts\/288"}],"collection":[{"href":"https:\/\/anvil-of-time.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/anvil-of-time.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/anvil-of-time.com\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/anvil-of-time.com\/wp-json\/wp\/v2\/comments?post=288"}],"version-history":[{"count":11,"href":"https:\/\/anvil-of-time.com\/wp-json\/wp\/v2\/posts\/288\/revisions"}],"predecessor-version":[{"id":1796,"href":"https:\/\/anvil-of-time.com\/wp-json\/wp\/v2\/posts\/288\/revisions\/1796"}],"wp:attachment":[{"href":"https:\/\/anvil-of-time.com\/wp-json\/wp\/v2\/media?parent=288"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/anvil-of-time.com\/wp-json\/wp\/v2\/categories?post=288"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/anvil-of-time.com\/wp-json\/wp\/v2\/tags?post=288"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}[file] & [file identifier table] >\n\tFile contains the data from source as recieved\/retrieved\n\tFiles stored in unique folder (date based) with unique name\n\tFile identifier table has simple structure:\n\t\tuniquekey\n\t\tsourceid\n\t\tdatetimerecieved\n\t\tfilename\n\t\tlocation\n\t\tprocessedflag\n\n\tProcess needs to:\n\t\tmonitor for files\n\t\tcreate destination location\n\t\tcopy file to destination location\n\t\terase file from source\n\t\tinsert record into file identifier table\n\t\tpossibly send acknowledgement back to file originator\n\t\tpossibly launch next step\n\n[raw data table] >\n\tHas single entry for each record in file\n\tSimple structure:\n\t\tuniquekey\n\t\tsourceid\n\t\tfileidentifierkey (uniquekey from file identifier table)\n\t\tdatetimeentered (when the record got into this table)\n\t\tdata (single record from file)\n\t\tpossible sequence number if multiples are sent from source in a single file\n\t\tprocessedflag\n\n\tIf the incoming data is many single record transmissions, you may want to eliminate\nthe file and file identifier table stage and simply put the messages here.\n\n\tProcess needs to:\n\t\tread file identifier table for unprocessed records\n\t\topen file\n\t\tparse file and insert individual records into raw data table\n\t\tmark record in file identifier table as processed\n\t\tpossibly send error back to file originator\n\t\tpossibly launch next step\n\n[transfer table(s)] >\n\tTemporary holding area for data being loaded into application.\n\tRecords marked processed removed prior to next load\n\tMay be several tables if raw data contains repeating segments (EDI\/Gentran type transmissions)\n\tColumn datatypes used\n\tTable structure can be complex:\n\t\tuniquekey\n\t\trawdatakey (uniquekey from raw data table)\n\t\tdatetimeentered\n\t\tsourceid\n\t\t(whatever columns are needed to hold data)\n\t\tpossible sequence number if multiples are sent from source in a single file\n\t\tprocessedflag\n\n\tChild or ancillary tables will need to have a foreign key relation to the master table\n\n\tProcess needs to:\n\t\tRemove rows marked as processed for same sourceid\n\t\tGet records from raw data file for the sourceid which have not been processed\n\t\tValidate datatypes\n\t\tWrite entry to error\/log table if a record is incorrect\n\t\tValidate 'completeness' of data (all required segments, length of data, etc.)\n\t\tWrite entry to error\/log table if a record is incorrect\n\t\tMark raw data table record as processed\n\t\tPossibly send errors back to file originator\n\t\tPossibly launch next step\n\n[application table(s)]\n\tData brought into the application\n\tProcess needs to mark the transfer table records as processed<\/pre>\n
\/\/get dw from pbl\nstring ls_PBLName, ls_data_object, ls_DWSyntax\n\/\/read stuff from ini file\nls_PBLName = profilestring ( is_ini_file, is_section, 'PblName', '' )\nls_data_object = profilestring( is_ini_file, is_section, 'DataObject', '' )\n\nIF ls_PBLName > '' AND ls_data_object > '' THEN\n\t\/\/ get the dwo from the .pbl\n ls_DWSyntax = LibraryExport ( ls_PBLName, ls_data_object , ExportDataWindow! )\n\t\/\/ create the datawindow\n dw_1.Create ( ls_DWSyntax )\nELSEif ls_data_object > '' then\n this.dataobject = ls_data_object\nELSE\n\/\/ whatever else you want to do.\nEND IF\ndw_1.retrieve()\n\/\/ and there you go...<\/pre>\n","protected":false},"excerpt":{"rendered":"