Yes, this is picture of my poor dog Buddy with a bow on his head. I thought it fit the headline well. I wanted to close out my thoughts on SOLIDWORKS World and provide a written account of my two presentations at the event. I also wanted to take the opportunity to thank SOLIDWORKS management for the opportunity to share my thoughts on data migration.

If you have read previous articles from me on data migration this one won't stray too far from the content of my previous writings but will be a fairly faithful summation of the presentation I delivered. I did put a different spin on some things and added a couple of specific case studies from data migrations we have previously completed and I will also discuss a couple of projects that are currently underway that have helped reinforce some of the best and worst practices I cover in the presentation. I did record a 30-minute version of the presentation if you want to hear it and see the slides as opposed to read through this article.

Migration Madness - Introduction

One of the reasons I chose the title "Migration Madness" for this presentation beyond the fact that it is cool iteration and you need a hook is that actively seeking out data migration projects requires one to be at least a little insane or maybe masochistic. Data migration generally presents unique problems and challenges not present in other types of PLM activities. The critical nature of this process and the intricacies of the planning required to successfully pull it off make it one of the more difficult activities in the PLM implementation field. In this presentation I discuss Zero Wait-State's history as a data migration solution provider and our qualifications to share our knowledge on this subject. I highlight a couple of key points about data migrations itself. I discuss best and worst practices for data migration and conclude with two case studies that feature moving information from Parametric Technologies' INTRALINK and PDMLink into SOLIDWORKS' Enterprise PDM. I also sprinkle in some lessons learned from current projects that are underway. As you will see from the graphics I tried to keep the slides basic and put the content in verbal form. Too many presentations at these types of events become eye charts crowded with bullet points which the speaker then dutifully reads to the audience. This approach has famously become known as "death by PowerPoint". I wanted to try and keep things interesting with entertaining graphics and use analogies to communicate the challenges surrounding moving data from one system to another.

Migration Madness - Zero Wait-State History

The first important fact I highlighted about Zero Wait-State is that data migration is part of our DNA. Most technical people are familiar with the term "zero wait-state". When we founded our company ten years ago we wanted a name that reflected what we do. Too many companies in our industry have meaningless initials or names that have nothing to do with their activities. I have difficulty sometimes recalling some of these names and I wanted something that would resonate. The term goes back to the early days of electronic design when memory chips and system busses functioned at different speeds. Wait-States were used to enable the faster system to wait and synchronize with the slower system. Eventually engineers developed "zero wait-state" memory which fully synchronized with the bus and provided optimal performance. We see similar parallels in the product development environment where technology and process tend to operate at different speeds. It also manifests itself in the different systems companies use to capture product development and manufacturing information. We develop software and provide services to eliminate these wait-states between process and technology and between disparate systems which results in optimal performance for product development organizations. Because of this focus we have pursued data migration as a core offering from day one and have had the opportunity to work with a significant number of companies and systems to provide data migration solutions.

Migration Madness - Partner Neutral

The second relevant fact about Zero Wait-State is that we are vendor and partner neutral. The formula on the chalkboard in this slide is a neutral element and that is what we are in the PLM and CAD ecosphere. To be successful as a migration expert and system integrator you need to be familiar with many different applications. Our history as a PTC reseller and our current relationships with SOLIDWORKS and Oracle have given us a broad spectrum of exposure to different technologies in this space. We specifically focus our energies on providing our clients with expertise in moving information from one system to another. We do offer general services for PLM implementation and upgrades and we have significant expertise in engineering collaboration but this capability spans multiple systems. We also tend to work with partners due to our tight focus so being able to support multiple platforms is a useful skill in this market. We typically do not resell CAD and PLM software directly but we do have relationships with partners to offer turnkey solutions if required. The critical point for this slide is that we offer a broad set of skills across multiple platforms when it comes to data migration and integration and are very principled about providing services to make all these platforms successful based on the needs of the client. We try to avoid becoming involved in advocating the advantages of one platform over another although we will be quite candid when it comes to diagnosing the best fit for a company's product development requirements. Because we work with many vendors and partners who potentially compete with each other we are very scrupulous in our rules of engagement.

Migration Madness - Zero Wait-State Experience with Data Migration

The final piece of information about Zero Wait-State is about our experience with data migration. Because of our focus in this area we have had the opportunity to work with numerous companies on these types of projects. Our customer list includes well-known companies like Dell, Raytheon, John Deere and Hologic. Even companies like these with large IT and support staffs are loathe to invest in training their resources for data migration. Data migration is a periodic activity that, if you are lucky, only occurs every four or five years so it doesn't make sense to expend energy spinning up indigenous resources to support this activity. Because we focus on this activity we have become adept at the process and tools necessary to move data from system to system so even though we are a fairly small company we are sought out for our experience in this area. Working with companies like these gives us the opportunity to deal with very complex systems and large data sets and to really understand the limits of technology and systems. We use this experience to develop best practices for how to feasibly move large data sets from one environment to another while adhering to the inevitable time constraints that accompany data migration. We also have had the opportunity to deal with data integrity and complexity issues that typically plague data migration efforts and to develop tools and strategies to identify these issues and resolve the problems in the most efficient manner possible. Beyond just data migration we have developed an application framework called DesignState which we leverage to tie disparate systems together. This activity is far more challenging than data migration since you must maintain ongoing connectivity between the two systems versus just having to move the data over one time. The experience we have picked up from developing this application has proved very useful for data migration. We also have been very fortunate to have companies like Cisco, Harris Corporation, Paccar and several of the companies mentioned above purchase our DesignState architecture to tie applications like PDMLink and Agile together to address their process requirements. We also are one of the few companies that has successfully migrated information from INTRALINK and PDMLink into EPDM which I suspect is why we were invited by SOLIDWORKS to present this year at SOLIDWORKS World. The end result is that by working with excellent companies, we have acquired extensive knowledge about successfully migrating data in and out of PDM and PLM systems.

Migration Madness - Data Migration is Hard

Now I want to shift gears and discuss a couple of truisms about data migration. The first one is that it is a really difficult thing to pull off successfully. The most apt analogy I have heard about data migration came from an IT resource at Dell in a meeting we were having with them concerning data migration. He compared the migration to changing out an airplane engine while the plane was in flight without crashing. If you think about it this really fits. One of the main reasons data migration is so difficult is that you are typically dealing with production systems. It would be pretty difficult to convince management at a company to shut down for a couple of weeks while you move their data. With the volume of data exploding it is becoming more and more challenging to move information over a weekend without disrupting a production environment. This means you have to create strategies to move and validate information in parallel with production environments which requires additional resources to support. The other challenging aspect of data migration is the data itself. It is very difficult to anticipate the challenges the condition of the data will present until you actually populate the data into the target systems. You can perform all kinds of due diligence ahead of time to try and assess the quality or condition of the data but invariably once you start importing the data into the new system the real fun begins. There is also a dependency on the customer since they are the ones that really understand the data. You would be surprised sometimes and how little companies know about their own data. Sometimes they don't know how much there is or where it is and they certainly may not realize they have broken links and ghost objects. While it is a good idea to perform your due diligence when it comes to data migration it is really difficult to fully understand all the nuances associated with the conditions of the data itself. The bottom line is data migration can be as difficult or more difficult than PLM implementation itself and should not be minimized or underestimated.

Migration Madness - Data Migration is Critical

The second often overlooked fact about data migration is that it is very critical to the success of a PLM implementation. Data migration is a taboo topic among PLM vendors. It is not fun to discuss the challenges of moving legacy data from one system into another. It is difficult to scope and the logistics can be daunting so best not mention it until after the deal is closed. I have actually received calls from vendors about trying to fit in a data migration after the sales and service purchase orders were placed. Minimizing or ignoring data migration is not a good idea. It doesn't matter how cool a PLM system is, how intuitive the interface is, or how many awesome productivity enhancing features it has. Without data in the system it might as well be a blank web page. Data is what makes the system productive. Hopefully, good, clean data that is accurate and reliable. If the data is bad or inaccurate or just plain missing it effects the adoption and utilization of the system. Moreover, product data is the lifeblood of any product development company. This information represents years of hard work and captures the intellectual property of the company. Without this information a company becomes a collection of people that can leave at any time and maybe a few buildings and rapidly depreciating assets. It is imperative that this information be safeguarded and given the priority and consideration it deserves. Without accurate product information in your PLM system your company will be running on empty.

Migration Madness - Best Practices: "Big Bang"

We can now transition to best and worst practices for data migration. When a company decides to change systems, there is often a psychological need move forward as quickly as possible. This might be analogous to ripping a Band-Aid off quick as opposed to slowly removing it. The idea is to get the pain of change over quickly as possible. Sometimes there are legitimate reasons for haste. A company could have a legacy system that has been off maintenance for quite a while and they are one failed hard drive away from losing access to their system or they might lack the infrastructure to run two systems simultaneously. Some companies don't want their employees to have to operate or support two systems simultaneously. All of these are good reasons to want to move off of a system in one quick move but with today's virtualization technology it is a simple matter to provision a virtual machine with whatever operating system you need and run it anywhere. The problem with using the "big bang" approach is that it creates enormous amounts of time pressure which can reduce the ability to filter and test data and jeopardize go live activities. One of the most successful data migrations we have been involved with was at Dell several years ago. At the time Dell was moving from one data management tool to another across multiple product groups. Each product group had some degree of autonomy particularly when it came to product releases. No manager in his right mind would want to be involved with data migration while releasing a new product to market. Unfortunately, there was no single time when all of the product groups would have a window to migrate over Our only choice was to create a series of scripts that would allow each group to migrate over when they saw fit. We moved the libraries and common use data that would not change into the new system and each group moved over when they were ready. The end result was that they were able to be more discerning about the data they moved and could make sure they were comfortable with everything before switching over. The groups moved throughout the year and by the end everyone was on the new system without any drama. The data they moved over was very clean and current and the new system performed very well for them. By moving over gradually it allowed them to scrutinize things more closely and ensure quality information was going into the new environment. Sometimes it is inevitable that everything has to go at once but if you can find a way to segment your data or move it over ad hoc as needed you will find the migration goes much cleaner.

Migration Madness - Best Practices: History is Best Forgotten

The next best practice is that history is best forgotten. George Santayana wrote that, "Those who cannot remember the past are doomed to repeat it". Obviously, George was never involved in data migration. The saying for data migration would be, "Those that forget the past get to keep their jobs." When migrating data less is more. Some companies have no choice in bringing older data over because of their industry or business model but most companies rarely access data that is in previous versions. One of the challenges with history is that legacy and target systems may lack the capacity to import and export revisions and versions or older workflows. Another problem is that some older data was created prior to PDM and PLM systems and may lack the structure to be useful in a modern system. The most basic issue is the sheer volume of information you are moving. This hinders the ability to effectively test and filter to ensure the quality of information going into the new system. In the end requiring history adds significant complexity to a data migration project and given how infrequently the data is accessed it may not be worth the extra time and money. If at all possible keep the legacy system in sustaining mode with a virtual machine and access the data as needed. You may be surprised how few times anyone goes back to the old system.

Migration Madness - Best Practices: Garbage In/Garbage Out

The next best practice slide continues to build on the theme of reducing the amount of data you migrate. Obviously, the easiest migration is one that involves no data at all but that is not going to happen. The key thought in data migration is just because you can move all your data doesn't mean you should. Often you are bringing problems that plagued your previous system into your new one. It is important to take the time prior to beginning migration to analyze your data and determine if it is accurate and worth moving over. It's about quality over quantity and ensuring that you do not pollute your new system with corrupt and inaccurate data. This can undermine confidence in the new system and affect performance. It is also about taking the time up front to assess and understand the condition of your data. This leads to our next best practice slide.

Migration Madness - Best Practices: Be Prepared

This summer we drove from Durango Colorado to Ouray. As you can see from the picture the roads are a little curvy. Fortunately, there are signs to prepare you for this so you can slow down and avoid careening off the mountain. The same principle applies to data migration. It is important to read the signs and slow down to avoid a crash and burn scenario. Preparations in this case include making sure you have sufficient hardware to process the large volume of data you will most likely be dealing with. In most cases, data migration processes often require that files be opened up in CAD systems to capture all references. Sometime this process can require data to be opened up multiple times to successfully capture all relationships so you really need robust equipment for the activity. Systems with fast hard drives, lots of memory and high bandwidth network connectivity are essential for ensuring smooth data migration. This is often overlooked and many companies scramble to find surplus hardware to use for the migration and testing which can really cripple a migration project. Another key for preparation building on the previous slide is to really analyze and clean your data prior to the project. It is pretty surprising to see how unfamiliar most companies are with their data. We have been involved in projects where companies have no clue how much information they have or what condition it is in. Before you embark on a data migration project you should allocate resources to go in and familiarize themselves with the data and randomly sample it for quality purposes to try and minimize issues later. The less data you move the better. Another point is don't wait until you are in the middle of the project to try and create scripts and tools for migration. It is very challenging to have to test and develop solutions in the middle of a project. If at all possible, try and plan for tool development prior to beginning the actual migration.

Migration Madness - Best Practices: Testing

The final best practice I want to address is testing. It is imperative that testing be structured into a data migration plan. Ideally, company resources are testing and sampling prior to the projects onset but it is more important to have a written test plan in place for validating the data once it's been moved to the target environment. In most cases a migration will move the data into a test instance of the target environment where it can be assessed. The assessment should be a thorough systematic methodology that is documented and followed faithfully. If any issues are encountered then additional passes can be done and measures can be taken to correct any errors prior to the final production migration. There is a temptation that once you have sampled a few files to truncate testing. Please resist this temptation. Many companies are not comprehensive enough in their testing and discover later that information is either missing or came across incorrect. The other point here is that companies need to test the data themselves. Consultants do not know enough about the data to determine if it is accurate. We can tell you we pulled 1500 files out of the source system and that there are now 1500 files in the target system but that is about it. Only you will know if the information is accurate in the new environment.

Migration Madness - Case Study #1

We will now shift to a couple of examples of data migration projects we have successfully completed. The first project involved a company, MDS Analytical, that wanted to move from PTC's INTRALINK 3.4 to SOLIDWORKS EPDM. As many PTC clients are aware INTRALINK 3.4 is the last release of the former PDM only INTRALINK tool and that all future versions of INTRALINK are based on the Windchill PDMLink platform. Upgrading to newer versions of INTRALINK requires a data migration and retraining for end users and administrators. More than likely the current infrastructure will be inadequate for the new version of INTRALINK and will need updating. In effect you are moving to an entirely new application. This fact and because the company was transitioning more to SOLIDWORKS as a design tool drove the decision to use EPDM instead of INTRALINK going forward. Keep in mind that managing Pro/Engineer data in EPDM is certainly possible but can be dependent on the specific use cases of individual companies. GoEngineer, a leading SOLIDWORKS reseller had engaged us for this project. They have developed scripts to import meta- data into the EPDM environment and provided us with formats that we needed to adhere to when we extracted the information out of INTRALINK. MDS had determined that they only needed latest versions which made the project much easier. We also were required to extract the binary data and to put it into specific file folders that could then be dropped into the EPDM vault. Overall this project went very smoothly. Being able to preload the meta-data really streamlined things as well as only having to move the latest revision. The client was able to move everything they needed and then shut down the legacy INTRALINK servers.

Migration Madness - Case Study #2

Bloom Energy was little trickier regarding the complexity of the project. Again, we were working with GoEngineer but Bloom was running PDMLink and using SOLIDWORKS for CAD. The project consisted of pulling PLM data out of PDMLink and moving it to Agile PLM and extracting the SOLIDWORKS data for migration into EPDM. The tricky part was that we learned the export commands in this version of PDMLink did not perform as advertised and we were forced to create our own query scripts to pull the information. This is where knowing things ahead of time would come in handy. We were able to turn around the scripts fairly quickly but it did cause a few days of angst for our client as we improvised to overcome the previously undetected obstacle. Query scripts are not the same as going directly to the database tables. This approach is highly discouraged. It typically invalidates warranties and since database schemas are not typically published you are at risk of not get accurate information. Once the scripts were in place everything again went smoothly. GoEngineer was again able to use their import tools to load the meta-data and obviously since we were moving SOLIDWORKS binaries the file load was very clean. The end result was that barring some difficulties this project accomplished the original objectives. There are several ongoing projects we are working on and while I can't go into great detail about them one of the lessons we have learned from these current project is that it is generally better if you migrate from some sort of PDM tool as opposed to a system drive. We have also experienced first-hand the consequences of under provisioning hardware for migration and not doing sufficient due diligence up front with the amount of data and its condition. We have also benefitted from lessons learned from previous migrations which allowed us to anticipate certain issues ahead of time which is much better than the alternative.

Migration Madness - Conclusion

The main points to take away from this presentation relate to not overlooking data migration and understanding how integral it is to the PLM adoption process. While data migration does have a reputation as being challenging, preparation can impact how well or poorly a project goes. Spending time up front analyzing the data and being selective about what gets moved into a new environment is a major factor in the likelihood of success. Another key component to success is the ability to segment data and move it across over an extended period of time as opposed to trying to move it all in one "big bang" approach. Finally, beware of underestimating the difficulty of data migration. It can appear to be straightforward, but as we have discussed, the condition of the information varies greatly and moving information while systems are still live is always a perilous exercise. I hope this presentation will be helpful for you. Please feel free to contact us with feedback or questions.

[Edit: repost from 2011]