Migration is one of those words that conjure up fear in the hearts of IT professionals. Migration is a word associated with hardship and danger when referring to the migration of people. Data migration is no different especially when Product Lifecycle Management (PLM) tools are involved. The best description for PLM data migration I have heard was in a meeting with Dell. One of their IT resources described it as "changing out the airplane engine while the plane is still flying without crashing the airplane. I think this is a pretty apt description. Migration can be an afterthought in a lot of PLM implementation projects but it definitely can dictate the success or failure of PLM and if done improperly it can result in a dramatic loss of altitude ending with an equally dramatic deceleration event as you hit the ground. Zero Wait-State specializes in data migration and we have spent the last decade migrating companies both large and small off of a variety of PDM and PLM platforms. This blog will highlight some of the perils and pitfalls around PLM data migration and best practices that can keep you flying high.
Garbage in Garbage Out
One of the first mistakes companies make when embarking upon a data migration project is assuming their current data structure is worthy of migration. Over the year companies accumulate a large amount of information and not all of it is critical to the company going forward and not all of it is in a condition that will make it useful in the new system. This would be a good time to take a hard look at your legacy data and determine what really needs to come over into the new system. Older product information may no longer be relevant and some information is not structured in a way to be useful in a PLM system. Obviously when you are moving from a PDM or PLM tool into a new PLM tool you are able to capture useful metadata like attributes but many companies migrate from system disks to PLM which can be much more challenging. Time needs to be spent evaluating current data and how relevant it is for future product development needs. Also if data is corrupted or sub optimal you should think hard before polluting the new system with the data.
2 Gallons of Water in a 1 Gallon Container
This is more of an issue for companies who are downsizing or as they like to say "rightsizing". Companies that elect to move from a more sophisticated PLM to a more affordable solution or an easier to use system can run into issues where they have more information to move than the new system can accommodate. Examples of this might be moving from Agile Advantage to Arena or from Intralink or PDMLink to Enterprise PDM(EPDM). The source system might support more data type or attributes than the new system can deal with so you need to take this into account when preparing the information to move over. More appropriately you need to consider this before you make the decision to move. If any of the information is critical you may be in for an unpleasant surprise. Workflows, Change history, file attachments, and access control logic are all things that may not be accommodated when moving from one vendor to another.
What a Tangled Web we Weave
It is not unusual for a company to want to migrate multiple PDM's or data sources to a single environment. While I certainly see the value in doing something like this it adds significant complexity to a data migration effort. Issues like naming conflicts and data integrity can severely hinder a migration involving multiple data sources. Coordination between the sets of information is essential and in depth analysis up front will save major issues from appearing after the migration is complete. Most PLD and PDM systems will not allow non-unique names so it is best to work out these issues prior to migrating into the target system. The more consolidation you can do prior to migration the less effort you will have to expend during and after the migration to the target system.
Volume, Volume, Volume
Some of the projects we have worked on involve massive amounts of data. I am sure you are aware of how much information even a small company can accumulate over time. This creates some challenges when it comes to migration. The first challenge is being able to convert all of the information in a timely manner. One of the keys to being able to change out the airplane engine in mid air is being able to do it at a time when no new information is being added to the source system. Large data sets can take several days to process so this can be a problem. There are two ways to resolve this issue. One is reducing the amount of data through purging or deletion the other is to migrate in phases. Phased migration makes a lot of sense for most companies because it allows you to break up the activity into manageable pieces. It also allows for validation and testing as you go. The idea is that you take periodic snapshots of the data you are moving and test the data to validate the process. Once the data has been validated you can do a final migration of just the delta between when you took the snapshot and the most current version of the source data.
History is Best Forgotten
There is a famous quote from the philosopher George Santayana saying, "Those who cannot remember the past are doomed to repeat it." George obviously has never been involved in data migration projects. History from PDM and PLM systems adds a degree of complexity to a migration that can be prohibitive. Extracting and capturing history from legacy PLM and PDM systems is very challenging. Ensuring that this information gets accurately transmitted into a target system can be even more problematic depending upon the sophistication of the import utilities. Using a "latest only" approach dramatically lessens the complexity of the migration and most companies discover that their history wasn't that useful. If it is absolutely necessary to preserve the history some companies will elect to keep their legacy system around or virtualize it and let people access it on an "as-needed" basis. It turns out for most companies that the need is surprising less than they expected. This also addresses some of the issues I mentioned above related to "garbage in garbage out" and "volume, volume, volume".
Be Prepared
The famous Boy Scout motto applies to many things but especially to PLM data migration. Preparation starts with hardware. It is a big mistake to under provision servers for this type of effort particularly if you are planning to move a lot of data. The other thing to be prepared with is a good set of data analysis and cleaning tools. As we discussed above understanding the condition of your data and being able to automate some of the cleanup can be very impactful. In most cases particularly with metadata there are scripts and applications to detect naming conflicts and to validate data transmission on the backend. It is important that you either have these capabilities or work with someone who does. You need to walk through the analysis process and understand the condition of your data so you are not caught off guard by the level of cleanup you will need on the back end of the migration. Ideally you should do as much clean-up as possible prior to moving information into the target system. It is infinitely easier to clean up information prior to migrating into a new PLM tool.
Test, Validate, Rinse Repeat
It is almost impossible to test too much during a data migration. We recommend two test passes minimum which should allow you to go into a test system and thoroughly sample the data before eventually moving to production. A well considered test plan should be included for the project either developed internally or with a partner's assistance. Ad hoc testing is better than nothing but there is a good possibility that you could miss something. Generic test plans are better than ad hoc testing but again without tailoring something specific for your environment you may find out the hard way that something is missing or wrong. If your company is required to validate per FDA or other regulations you can combine some of this effort into testing and potentially compress your testing cycle. It is best to get testing addressed up front before the project gets too far down the road. The temptation to truncate testing is great so you must resist.
The End of the road
Most of these ideas are common sense but surprisingly often are disregarded in data migration efforts. Too many times the migration project is just bolted on to a PLM implementation as an afterthought and in the rush to get the new system into production the migration discipline is compromised. It is important to recognize that PLM tools are just vessels that hold the most important asset a company has; their intellectual property. It is far more important to ensure that information is transferred cleanly and accurately than it is to stand up a new PLM system. Plane crashes are tragic and obviously anything that happens with migration pales in comparison with the loss of life experienced but in the context of business nothing can be more crippling than losing product information. This data is the lifeblood of a product development company so safeguarding it should be one of the highest priorities a company has.
|
Comments
Very true. Most companies don't work with data much older than 2 or 3 years so they don't really know the condition of the data that is older and that the point. They rarely access it so why go through the cost of migrating it when its going to be so expensive and time consuming. However, even moving history that is clean and recent can be challenging depending on the systems involved. "Latest only" is definitely the way to go if it is at all possible.
Well written and so true. I have got to save this for the next migration. When the dreaded "legacy data" creeps into the scope discussion.
I am usually trying coach the legacy data out of the data migration scope (if possible). This helps that discussion with good talking points. Thanks!
Nice work
Respectfully,
Bradley Persson
Thanks for the kind words. This blog seemed to strike a nerve with a lot of people. Migration is never fun but as they say "its a dirty job but somebody's got to do it". Let me know if we can help if you are unable to convince clients to forgo moving the data.
I think the most important doctrine to remember is, move only those things that will continue to have hungry consumers to either build or refine something. Rest is best forgotten.
Many organizations have this tendency to create artifacts that invariably are write only with no consumption possibility. This only lends a helping hand to delay development life-cycle because of the unnecessary process steps and work effort that could be possibly used elsewhere more productively.
I always thought that a prudent approach would be to move consumable product record data based on product family, one at a time. To me the problem of data migration is like classical set theory. You have got to understand the Venn diagrams of your engineering data before you embark on the treacherous journey of migration else you will always be tempted to move a whole lot of garbage that will either rot or contaminate your useful data.
For some reason your post has me craving lunch. You are right on the money about understanding data before you move it. This probably should have been a point I included in the article. It is sort of related to garbage in garbage out but a bit more involved. Most companies don't understand their data and are reluctant to leave any behind in case they discover they need it in the future. Spending some time with the information before you migrate would also help you better understand how you should create information in the future. By just blindly pulling your information from system to system without any analysis you end up repeating the same errors and building a chain of mistakes similar to the one Jacob Marley drags along in the Christma Carol.(seasonal reference, bonus points).
Well done but I want to extend this a little bit because the migration problems occur even when you are upgrading your PDM or PLM.
In 2005, I worked as a PLM Consultant where a very few of us were upgrading Teamcenter Enterprise for Mattel. This required a full migration because they were bypassing several releases and the base data model and methods APIs had changed considerably. Not only was the data different inside the PLM environment, there were external programs accessing the bulk data without using the API at all. To complicate things even more, the system had first been assembled with replica databases all over the globe but they now were going to consolidate into a single Oracle server.
The various programs had been implemented in many technologies and none of those technologies were using the same environments. I mean the Bourne shell scripts used one set of environment variables, the Perl scripts a different one, the Java code used properties instead of environment variables, and the C language methods most often used hard-coded data. Duplications, erasures, mistakes, everything that could be wrong was wrong. Mr Murphy had won the game for years.
With my years of experience developing commercial product for that PDM, I needed 9 months to rewrite the tools to use a set of variables and properties that would be maintained in a single file installed in the configuration. Because the HTML files they used as their bulk data had been placed in a different hierarchy than the new target would have, a massive undertaking was to use UNIX tools, sed, csh, cut, paste, etc., to remove and replace the hard-coded pieces with generic values from the runtime.
We did it right, though. We didn't need another iteration. Everyone had worked very hard and most of the work was done by experienced people who'd seen so much of this before and knew how to fix the problems. It took a long weekend to do the final installation and validation but we were ready and weren't disappointed.
So, migration can happen to anyone at any time.
RSS feed for comments to this post.