High Volume GP Integration Tip: Increment your batch ID every X thousand records

By Steve Endow

I've developed several high volume Dynamics GP integrations for customers that needed to import tens of thousands of transactions at a time, and I also work with customers who use Post Master Enterprise to automatically post their batches--many of whom have very high volume integrations.

While the process of importing 20,000 or even 100,000 transactions into Dynamics GP is not terribly different than importing 100, there is one thing that can be challenging with such high volume GP integrations:  Batch Posting.

eConnect imports can shove a lot of data into Dynamics  GP very quickly, but once the import is done, you are stuck with a massive pile of transactions to post in Dynamics GP.  And those batches can take a long time to post.

One thing I always recommend to customers who have to import thousands of transactions at a time is to have their import break up the data into multiple Dynamics GP batches.  So instead of having a single batch with 40,000 transactions, have eight batches with 5,000 transactions, or even 20 batches with 2,000 transactions.  There is no magic number for batch size, but I would personally keep them under 5,000 per batch.

I recommend this because if there is a batch posting failure or an error that causes the posting to stop, you won't be holding up 39,000 transactions.  If the batch size is 2,000 transactions, a posting failure will only delay posting for those 2,000 transactions, while 38,000 transactions in the other batches can continue to post.

So how do you split your import of 40,000 transactions from a single data file into multiple batches?

I'm glad you asked!  It's quite easy:  You just change your batch ID.

In your import, you will maintain a transaction counter that gets incremented every time a transaction is imported.  After that counter hits X thousand records, you change your batch ID and reset your counter.

I prefer to make the batch size a configuration option in my integrations.  So the customer can change it to any value they want at any time.  Then, when I assign the batch ID to the transaction, I check the counter.  If the counter hits 5,000 (or whatever value is configured), I increment the batch ID suffix, like INV12345-1, INV12345-2, INV12345-3.

If you have a .NET eConnect integration, this is just a few simple lines of code.  If you are using SmartConnect or Scribe, you would probably need a few global variables to store your batch size and transaction count, and a few lines of script in your Batch ID field.

To me this technique of managing batch size seems very obvious, but I have had dozens of phone calls with customers who are importing tens of thousands of transactions into a single batch, and then stressing over the batch posting process at 1am, because a failed batch posting would take them hours to re-post.

Skip the stress--split your batches.

Happy importing, and merry posting.

Steve Endow is a Microsoft MVP for Dynamics GP and a Dynamics GP Certified IT Professional in Los Angeles.  He is the owner of Precipio Services, which provides Dynamics GP integrations, customizations, and automation solutions.

You can also find him on Google+ and Twitter

http://www.precipioservices.com