IBM MDM Suite

MDM product from IBM is known as InfoSphere Server. For web they have WebSphere and for Data (Information) obviously InfoSphere :).
According to IBM site this product provides MDM services for various type of domains :- "customer, account, product, as well as other data types like location".

They have a product called "IBM InfoSphere Master Data Management Server for Product Information Management". So this is actually PIM but they call it "a core element of a customer’s master data management strategy".

According to IBM website InfoSphere Master Data Management Server: "contains more than 800 business services out-of-the-box to manage both complex and simple master data inquiries and updates."

One thing which looks interesting is IBM's strategy of going multi domain in MDM: all types of master data, whether it is Product or Customer data.

Oracle Master Data Management Suite

MDM is essential for success of SOA. It requires tools to achieve its goal of single, standard, common view of master data. Almost all the vendors are offering MDM tools and software. Let us see what Oracle is offering in this space

Oracle Customer Hub
Customer data integration (CDI) solution. It provides following:
A prebuilt extensible global master data model for customer information
Services to maintain customer data across various systems and processes
Capabilities to acquire, synchronize, and federate data
Change and audit controls for data governance

Oracle Product Hub
To centralize all product information from heterogeneous systems. It provides:-
Import workbench for improved data quality
External transaction views.
Mass maintenance capabilities to manage updates to multiple products
Product sales configurations for manufacturing and service industries
Change control capabilities
Rich user interfaces for unstructured content management
Predefined operational attributes with unlimited extensibility

Oracle Site Hub
It is a location mastering solution. It provides:-
A prebuilt extensible data model (site master).
Unlimited number of predefined and user-defined attributes for consolidating site-specific information
Manages updates to multiple sites
Site mapping and view using Google Maps®
Prebuilt integration with Oracle Inventory, Oracle Property Manager, and Oracle Enterprise Asset Management.
Web services to share data and integration with various systems.

Hiperion data relationship management
Previously known as Hyperion Master Data Management. To build consistency within master data assets. According to Oracle website "it provides the industry's first data model-agnostic master data management solution built to enable financial and analytical master data management in dynamic, fast-changing business environments".

For more information please visit Oracle Website:- http://www.oracle.com/us/products/applications/master-data-management/index.htm

SAP Master Data Management

Master Data Management (MDM) product from SAP is known as "NetWeaver Master Data Management". NetWeaver MDM is part of (one of the component of) SAP's NetWeaver product suite.

SAP NetWeaver is foundation for all SAP applications. It is SAP's SOA platform suite. NetWeaver is built upon open standards and inter-operate with, technologies such as Microsoft .NET, Sun Java EE, and IBM WebSphere. It can be extended by Microsoft .NET of Sun Java EE.

Standard architecture for SAP MDM can have a centralized MDM server and clients connected it. Clients are connected to MDM server through SAP's Exchange Infrastructure using XML documents. Clients can also connect without Exchange Infrastructure.

Implementation scenarios

• Centralized cleansing, removal of duplicates and consolidation, key mapping and reporting in SAP BI. In this scenario data is not supposed to be sent to originating systems.
• Above mentioned activities along with redistribution of processed data to originating systems.
• Maintaining master data at a central MDM system.

SAP purchased A2i in year 2004. A2i was in to Product Information Management. It is believed A2i is capability and resource became foundation for SAP's MDM till version 5.5 and that is why some people didn't consider it as a full MDM but a PIM. Newer versions 7.x are better equipped to provide full MDM capabilities. Although adoption of SAP MDM was some what slow but we will have to see whether it will make any difference in its new version or before that SOA story itself will change its direction.

Book - SAP NetWeaver for dummies By Jeffrey Word preview with limited chapters is available on google books.

Sun MDM suite

(Note: This post was written long time back before Oracle acquired Sun so now we will have to look in to future strategy of Oracle on MDM)
The Sun MDM Suite is based on the Sun Java Composite Application Platform Suite (Java CAPS). Sun MDM is developed in open source. It includes an infrastructure layer, Enterprise Service Bus (ESB) layer, MDM layer, and portal layer.

The MDM layer includes the core MDM products. Following is description of these products:-

Sun Master Index
Defining reference data structure, storage and maintanance of this reference data.

Sun Data Integrator
For extracting legacy data from existing systems, transforming it and loading it into a master index database.

Sun Data Mashup Engine
Data aggregation from multiple sources to provide a single view of enterprise data.

The Sun Data Quality and Load Tools
Profile, cleanse, standardize, match, and load the reference data. It uses "Sun Data Integrator" for loading. Following are the tools related to data quality & loading:- Master Index Standardization Engine, Master Index Match Engine, Data Profiler, Data Cleanser, Initial Bulk Match and Load Tool

I have summarized Sun MDM suite for the benefit of readers here but I have not described them in more detail because a very good documentation is availabe at Sun's site. For more information please refer:-
http://developers.sun.com/docs/javacaps/products/sun-mdm-suite.jsp

What is MDM ( Master Data Management )

I would like to share my understanding of MDM from its requirement in an organization

Master Data Management

Management doesn't mean a technology all agree? Management is a practice, approach, anything but not technology. For the time being I am taking it as a  practice.
Master data means nothing but non transactional data. So what? Why am I worried about it. DBA's used to manage the database so master data too. Exactly.

Than what are the challenges of managing master data?
Now if an organization has various department, offices located at various parts of a country or even globe, in such a case even if all of them are part of a single organization and obviously they have data related to same organization but since they are located at different places and their functions are different they can have duplicate data, not matching with each other (in structure or definition). Inconsistencies may be there if master data which is common to whole organization is not defined and standardized and properly controlled, processed blah blah blah... So the challenges:- Consistency of organizations master data.

There should be a process to do following things on master data of the organization:-

• collect
• aggregate
• match
• consolidate
• assure quality
• persist
• distribute

Obviously tools will also be required to do above mentioned activites.

So MDM is a set of process and tools to manage master data of an organization. One important silver bullet- When we are talking about distributed department having their own copy of data, what does it indicate. Distributed environment requires silver bullet SOA :). So important to consider here is that MDM is very important for success of SOA.

Impacts of poor integration of master data

If organizations have 40, 50 and more databases and systems and data of the organization is not managed properly there may exist a situation where every system has its own definition of master data. Since master data is shared among various operations various issues operational issues may arise

Customer satisfaction
If you don't have a single view of a master / shared information there will be a communication issue. Say for an example in a mobile service provider company there is no standard shared master data of customer. If a customer has already discontinued the service, their billing department may continue sending monthly charges bills to the customer. This is my personal experience with more than one service providers.
This leads to dissatisfaction in customers. This is not only applicable to a scenario of discontinued customer, this is also true for customer who have already paid their bills but still get calls from agencies who work for service provider.

Operational efficiency
What a provisioning department or billing department will do if it has different view of customer data. Obviously efficiency of the overall system will be impacted.

Decision-support
How can you trust a piece of information if you know what ever it is showing may not be correct. Something exactly same exist with different name of definition;. In such a condition how decision support system can work.

Regulatory compliance
Inconsistency may create issues related to regulatory compliance. You don't have a single definition of lots of things in the organization and hence you will produce wrong facts and figures in front of regulators. This can lead to compliance issues.

Management of master data becomes more challenging in acquisitions and mergers scenarios where two organizations have their own data but ultimately it is going to be a single organization so some data must be common and shared.

MDM (Master Data Management) Process

What should be the process of Master data management? What are the activites which needs to be done? Let us examine this very quickly.

• Identification of source of data
• Collect Meta-data
• Collect Data
• Create Master Data Model
• Select appropriate tools
• Transform
• Normalize
• Apply Rules
• Data correction
• Generate and Test Master Data
• Change producer or consumer if required

Following will run along with above activities in MDM journey.

Data governance
Body to set the rules, decide standards and monitor MDM process.
Data stewardship
Who will run the MDM projects for their own departments. Owners of data/information

MDM - Approaches to maintain a Master List

There are many ways to maintain master data. Specially when it comes to maintain master data in an existing system where there are number of applications ad they have their own data along with copy of master. I am trying to summarize various options to maintain master list here. In my future posts I will again revisit approaches.

Only one copy of master— Single master data. Every change is directly made to the master data. Change in user of master data applications is required to use new data. Benefit of this is that you have consistency but it is not always practical.

Many copies, one maintenance— Master data added to the single copy and changes sent to the systems where copies are stored locally. These applications can change the data which is not part of master. Benefit if this is that it requires minimal in the applications.

Merging Master — Applications can change their their copy of the master data. Changes sent to the master where changes are merged into master. These changes are sent back to source systems to update local copies. Benefit of this approach is minimal changes in the source systems. There can be issues like:- Conflicts of same update by two systems, adding a new item in one system where other system already have that item will require remerge. Matching process to avoid duplicates in the master

MDM - Styles to manage master list

There are different sets of data in an organization like product information, customer data and location data. Any MDM product or solution may use various hubs to manage these sets of data. The hubs can manage their data in any of the approach (I have explained in my previous post -one copy, multiple copies etc). Here I am putting some more thoughts on styles of master data management:-

Single Copy Master (Central Hub)- A single database for master data with all the information required for applications use it. After linking and matching data is consolidated at a central hub and published to various data sources. Central Hub prevents duplication. Currently running application may require changes to use master data.

Many Copies one Look-up Service- Master data is maintained within user application databases. The central hub stores lists of keys. These keys are used to access attributes for a an entity and these attributes are linked to databases. To access master data, data is located, a query is distributed among various databases, and a list of the requested data is formed on the fly.
This style doesn't require many changes in existing systems but in case of huge databases and addition of more and more databases this can become too complex for efficient access of data. Another issue I have already mentioned in my previous post is chances of duplicates in various systems. With addition of new databases queries may require changes.

Mixed style- Mix of both the styles. Master data is kept on the native databases, keys and Ids are generated to access this data. Some of its important attributes are replicated to the central hub. Central hub can service the common requests, and queries are distributed among various native databases specific attributes (which are generally not frequently used). This improves efficiency of data access at the same time requires less changes in the applications. Update of replica may have issues in this style. Identification of attributes, standardizing format can be a challenging task.

Role of Datanetworks, Datawarehouse & Datamart in MDM

Managing master data was one of the most difficult tasks for organizations. There are ways to solve this problem but tools can help managing master data across organization. Some of the tools are data networks, data warehouses, data marts and an operational data store. Now a days almost all vendors are offering tools for master data management. Here I will throw some light on few types of tools, what they are? what they provide etc.

Data networks
Data networks facilitate in transmission of data within an organization. They provide information transfer and storage facility. A common server can have data which can be used by different departments or even group companies of a corporate. With data networks it is possible to gather master data from various sources. Obviously this will require consolidation and management. Another function data networks provide is sharing of important business information which is used daily transactions.

Data warehouses
Provide storage for the master data. Data can be analyzed and reported through it. Along with storage of data they also provide processes for the retrieval, analysis and management of data for proper access. Since they use common data model and they are optimized for data retrieval they facilitate easy analysis and reporting. This way source system (involved in transactions are relieved of this task and can work efficiently without degrading performance of source systems. Datawarehouses provide long storage of data which may not be possible in transactional db systems.

Data Mart
Although data marts can exist independently but often they are consolidated to dorm a datawarehouse. In some cases subsets of data is extracted to create data mart. Generally data marts are subset of datawarhouse. More specific data (related to a particular aspect) is kept in data mart. Individual departments or group of users use data marts to access data related to their business requirements. Since they are focused on specific information, frequently used data access from data marts is easy.

Transactional Databases
They store real time transactional data.

Which MDM product is the best?

(Note : You can also vote on this blogs poll to choose best MDM product but I have also posted a detailed feature based survey at Survey Monkey)

MDM is becoming more important and relevant now a days for enterprises. There are tools, products which can help implementing Master Data Management but lots of vendors have launched MDM products in the market. Some are big players and some are small players but specialists. Which product to choose is a question which needs to be answered.

MDM Selection Criteria

Before choosing a product there are some criteria which needs to be prioritized for an organization and on the basis of these criteria a product supporting features in favor of the organizations requirements can be considered. Here are few things which can be considered:

• Support for Multiple-Entity types for example customers, accounts, products, etc.
• It should have support for requirements of various user types and usage scenarios (analysis, reporting etc.).
• Data governance and administration features and capabilities.
• Support for unstructured data and search capabilities for it at enterprise level.
• Centralised or decentralised repositories support.
• Ease of deployment
• Pre-built Domain Data Model

Comparison of MDM Products

To compare various MDM products I am conducting a survey through Survey Monkey to get feedback on these features to select a MDM product.
I request you all to vote in this survey. It is a single page survey and doesnt require too much of effer. Just click and finish. Please click here to vote.

This survey will provide enough ground for MDM product comparison. Which one is the best MDM? Answer to this question will depend upon what are the organzations priorities but these features will help them choosing a MDM product based on their priority.

MDM readiness assessment tool

TDWI & Baseline computing have designed an assessment tool for MDM readiness. According to TDWI website: "This interactive survey has been designed to help you gauge how prepared your company is to acquire an MDM solution and launch a sustainable MDM program"

It asks about 37 questions on various pages. Questions are based on your general understanding of your organization to figure out what people in your organization think about master data, what do you do to clean duplicate data etc. Based on these and other types of question it sends you a report to your email ID.

Important thing about this survey is it has questions touching various aspects of data:- Quality, Processing, Change Management, Organization, MDM Perception people have in your organization, Rules and Policies, Data Access & Navigation etc. Its report shows your score in various categories as compared to other organization which have undergone this survey.

Since its score is showing your scores as compared to other organizations score it can be useful to know maturity of MDM at your organization. But at the same time it can be erroneous because organizations participated can not be a scale to know your maturity.

Last month "Revised joint submission SoaML for UPMS" was submitted by some vendors to OMG. This was much awaited submission for me because I wanted to know what finally they are putting on table. How they have taken modeling for SOA is interesting to see.

It describes a UML profile and metamodel for service oriented architecture's service design. Their aim is to fit it into model driven approach. It is based on the UML 2.0 metamodel L2. Important thing here is that they say it conforms to OASIS SOA Reference Model.

Service is shown with "Pakage" notation. Every thing is covered but they have tried to accommodate everything using existing symbols and notations. This doesn't look natural for me. Specification can be found at http://www.omg.org/docs/ad/08-08-04.pdf

I doubt this is going to make life simple. I doubt this will be used and if will be used will help people. I was unable to see any other aspect except old object oriented aspect in this specification. I believe this specification is created to facilitate existing tools so that no change in them will be required. What I think from SOA is that it is entirely different from normal OO paradigm. Architecture and modeling of it should be in such a way that can be understood by business people more.

There are some basic architectural rules and principles which should be applied to a good architecture. All those rules , principles and best practices are also applicable to SOA based system design but there are best practices and principles which are specific to SOA because of specific nature of SOA. SOA systems have some specific features due to which design considerations should be special for SOA including basic design principles applied to any architecture.

Service Contract
Service should implement an standard interface. This interface is a contract with service consumers. One of the goal of SOA is re-usability across organization and another goal is integration. In this scenario there will be various consumers of the service across and beyond organization. If service will not adhere to a contract, service consumers will not be able to use service. One of the most important design principle of SOA is conformance to service contract.

Loose coupling
Service contracts should be designed in such a way that they should promote loose coupling between provider system and consumer system. Since consumer and provider of service may be located at various places under control of different entities tight coupling will have ripples effects which can not be handled easily in case of any internal change in service provider. As a good design principle services should have coarse grained interface to improve loose coupling.

Abstraction
Another good SOA design principle is abstraction. A service contract should only have limited and essential information.

Re-usability
A service should be reusable on its own. Here this SOA design principle is considered due to commitment of SOA to promote re-uase. In general architecture rules or principles you might not have seen specific focus on reuse. General architecture design principles advocate about loose coupling, cohesion etc but they talk less about re-usability. SOA promises high ROI so reusability has special focus in good SOA design principles. There are two SOA design patterns help here - logic centralization and contract centralization. applying these two SOA design patterns results in to standardized and normalized service which fully support and maximizes re-usability.

Autonomy
This is one of the most controversial claimed tenets of SOA. As there was no unanimous definition of SOA and SOA world this SOA design principle (Autonomy) also has several meanings and understandings. I have serious objections and I will start a debate on it later but for the benefit of readers I will not make it an issues at this place and would like to summarize few things
Bruce Johnson in his blog has discussed this controversial issue. He writes "A service, on the other hand, has a life outside of the calling application. If the calling application dies while the service is doing its think, the service will continue functioning properly. In fact, one of the guarantees of autonomy (in this sense) is the ability for a service to be called asynchronously with no expected return value. He further says "a service, where possible, should be self-healing". I support Bruce Johnson's views that it should not be called Autonomy. Autonomy is a wrong word chosen for expected behavior of a service mentioned by Don Box with respect to Autonomy. Autonomy means independence but what about scenario where a service will use other services to complete its task. A composite service. People have written that with loosely coupled, they are location neutral, services communicate through dynamically negotiated channels etc etc that is why services can be considered autonomous. I didn't understand why decoupling, standard contracts etc which are other design principles are used to establish this design principle (autonomy).
Yes services should be designed keeping these things in mind (location neutrality, they should not know how other is working, they should have standard interface but how does it make a service autonomous? In practice no service can be autonomous. As Bruce has written "My own personal choice would be 'sovereign'". I still don't want to call it sovereign.
Some explain service autonomy as "Service should have high level of control over its run-time environment.". This is funny? :)

Statelessness
There should not be long-lived relationships between consumer and provider of service. Any operation invocation should not rely implicitly on a previous invocation. Making service state full will have negative impact on availability and scalability of the services.

Discover-ability
Services should be easily identified and understood when opportunities for reuse are present. The service design should focus on “communications quality” of the service and its individual capabilities, regardless of whether a discovery mechanism for an example service registry is there or not.

Composability
Services should be capable of participating as effective composition members. Re-usability promise of SOA requires that new capabilities, business requirements can be fulfilled with no time by composing a new application using existing services. Good design effort is required to make sure it should promote composability.

Due to specific nature of SOA eco-system it requires a new model methodology and thinking for testing. Why? SOA services span across boundaries of the organizations, applications are composite of various services across the organization and beyond organization boundary.
For a long time there was a need to have SOA testing model specially designed for SOA but it didn't exist. Here I am trying to put some thoughts on SOA testing.


Challenges in SOA Testing

• Some services don't have user interface- they are used by other services from outside of the organization which participate in current services interact and collaborate with out side world beyond organization boundary.
• Security testing is utmost important and is a challenging task in such an heterogeneous and uncontrolled environment.

Testing Model

Famous V Model comes into picture here. Test design analysis planning and execution happens throughout the life of SOA project. V model provides for parallel testing activities during project life cycle. Example- Requirement=Prepare UAT, Functional Specs=Create System Service Test, Tech Specs=Create Integration Test, Program Specs=Create Unit Test

Why V model is suitable for SOA projects?

• Derived from business context.
• Starts from business requirements gathering till acceptance.
• It takes top down approach which helps capturing various aspects like processes, security etc.
• Various levels of V model map to various phases of project life cycle covering different level of details and various view points e.g. functional view point, integration view point, user view point

Requirement for SOA testing

• Detailed understanding of SOA
• Tools to cater testing requirements from business domain, security etc.
• Business domain knowledge and test teams aligned to domain.

Types of testing for SOA

• SOA testing can be categorized in to following
• Service component testing
• Service testing
• Integration testing
• Process testing
• System level testing
• Security testing
• Governance testing

I was trying to find out whether some one has done comparison of SOA tools and products in past or not. I wanted to share my thoughts comparing SOA products but I found people have already done it for you. Still I will do some homework and will come back with my comparison till then enjoy these resources on internet for SOA product comparison

Current Analysis

http://www.currentanalysis.com/p/BusApp-abstracts/AppInfra-SOA.htm

ENTERPRISE SERVICE BUS: PRODUCT EVALUATION COMPARISONS
http://dts.utah.gov/techresearch/researchservices/researchanalysis/resources/esbCompare061018.pdf

MyArch Solutions

http://myarch.com/comparison-of-soa-suites

Network Computing

http://www.networkcomputing.com/channels/appinfrastructure/showArticle.jhtml;jsessionid=CKJYQ2X3W1HY0QSNDBECKICCJUMEKJVN?articleID=181501276