Dramatically Improve Dynamics GP eConnect Performance, But There's a Catch

My blog has moved!  Please visit the new blog at:  https://blog.steveendow.com/

I will no longer be posting to Dynamics GP Land, and all new posts will be at https://blog.steveendow.com

Thanks!




By Steve Endow

If you've developed a Dynamics GP integration using eConnect and .NET, you should be familiar with the eConnect Serialization process, whereby the eConnect Serialization .NET assembly converts your transaction objects and data into an XML document.

Here's an example of the serialization of a GL transaction.

Notice that this code involves a single GL transaction object, glTrans, with the glTrxHeader and glTransLines assigned to the object properties.

That single GL transaction is then assigned as a single element to the glType array, which is then converted to XML.

Wait a minute.  Why is the GLTransactionType object, in this case named glType, an array?

Well, it's an array because eConnect Serialization supports the conversion of multiple transactions into a single XML document.

What?

So...why would you want to do that?  Why serialize several transactions instead of one transaction at a time?

Good question!

Here is a video where I discuss this feature and demonstrate the performance:

I asked that same question ages ago when I was developing with eConnect on Dynamics GP 7.5, probably just over 12 years ago.  I was using Visual Basic 6 and the eConnect COM components.

When I saw that eConnect serialization supported transaction arrays, I assumed there must be some reason why that feature existed, so I performed a test to compare the import performance.

I recall that with VB 6, when serializing and sending individual transactions, I was able to import an average of 20 AP invoice transactions per second, which is actually quite a bit better than what I see with .NET today.  And I recall that when I tested serializing and sending multiple transactions at the same time, I did not see a performance benefit. I was therefore puzzled about the functionality.

While I didn't observe any performance benefit, I did find a downside to sending multiple transactions to eConnect at once.  If eConnect encountered an error while importing any of the transactions in the 'batch', the entire batch would be rolled back and no transactions would be saved.  So if you send in 100 transactions, and 1 transaction fails to import, nothing is imported.

And another annoyance:  If your batch of 100 transactions had 5 transactions with errors, eConnect would only return 1 error.  As soon as the first transaction has an error, eConnect stops, rolls everything back, and returns a single error.  Once you fix error #1 and resubmit, eConnect will stop on error #2.  So you are forced to deal with the errors one at a time, and in the meantime, zero transactions have been imported.  It's not an ideal error handling process.

So, fast forward to the year 2018, when I was recently asked if eConnect could import multiple transactions in a single call.  I initially responded that it can, but that there was no performance benefit or value in doing so.

But then, for some unknown reason, I figured I should test single vs. multiple transaction submissions and verify my long-held 'belief'.

So I pulled up my eConnect load testing tool and modified it to support serializing multiple transactions, in addition to one transaction at a time.

When I saw the results of sending multiple transactions to eConnect in a single submission...

My...

Jaw...

Dropped.

100 GL JE transactions imported in 17 seconds when sent individually to eConnect.

But when I submitted a single XML document with 100 journal entries, eConnect imported them in 3 seconds.

WHAT???

I then imported 700 transactions, submitting them all to eConnect in a single import request.  They imported in 17 seconds, the same time it took to import just 100 transactions individually.

That's a 7 fold increase in performance, resulting from a minor change in the import code.  That is a staggering difference.

I repeated the tests several times and confirmed that it wasn't a fluke.

I then tested with AP Invoices and saw the same results.  In this test, I was able to import 750 AP Invoices in a single submission in the same amount of time as submitting 100 AP Invoices individually.  That is cra-zee.

So, based on my new tests with GL JEs and AP Invoices, submitting multiple transactions to eConnect can produce a huuuuuuge performance improvement with the eConnect .NET assemblies.

My guess is that something changed when the eConnect assemblies were converted to .NET, resulting in a slower performance when importing individual transactions, but faster performance when importing multiple transactions.

But....

There is a catch.

It's the same catch that I found when testing with VB 6 and eConnect COM objects.  If there is a single import error, the entire batch will fail to import, and no transactions will be saved.  So if you try to import 500 transactions, a single error will roll everything back and you'll need to fix the error before resubmitting.  And if there is a second or third error, the entire batch will fail a second and third time.  495 transactions will be held up because of 5 errors.  It's a hassle.

And if you assigned 500 journal entry numbers and the batch of transactions fails to import, you've just burned through 500 JEs.  When your auditors review the JEs in Dynamics GP, they'll see a gap of 500 JE numbers and start asking questions.  If such errors happen frequently, you'll quickly burn through thousands of transaction numbers.  It's not the end of the world, but it's not ideal.

So, given these drawbacks, what's the conclusion?

My personal recommendation is that most Dynamics GP users should stick with submitting single transactions to eConnect.  For most Dynamics GP users, eConnect imports are not a bottleneck for the business, and I find that it is uncommon for customers to be waiting around for eConnect integrations to complete.  Usually, customers are able to import hundreds or thousands of transactions relatively quickly, but are then having to wait for those transactions to post in Dynamics GP, as posting is far slower than importing.

However, there are some very high volume customers that import tens of thousands or hundreds of thousands of transactions every day, and are either importing or posting transactions non-stop throughout the day.  If you have Dynamics GP integrations that collectively take hours to complete, and this technique could save you hours of import time, then I would definitely recommend looking into it.  But only if you 1) have the ability to develop pre-validation routines for all of your transactions to minimize errors, and 2) have the ability to deal with errors and reimport the failed transactions, and 3) you can develop a way to manage transaction numbers so that they aren't wasted with each failed import.

I hope this was informative!

Steve Endow is a Microsoft MVP in Los Angeles.  He is the owner of Precipio Services, which provides Dynamics GP integrations, customizations, and automation solutions.

You can also find him on Twitter, YouTube, and Google+

http://www.precipioservices.com