Home > Architecture >
SOA & MDM
MDM products offering from vendors
IBM MDM SuiteAccording to IBM site this product provides MDM services for various type of domains :- "customer, account, product, as well as other data types like location". They have a product called "IBM InfoSphere Master Data Management Server for Product Information Management". So this is actually PIM but they call it "a core element of a customer’s master data management strategy". According to IBM website InfoSphere Master Data Management Server: "contains more than 800 business services out-of-the-box to manage both complex and simple master data inquiries and updates." One thing which looks interesting is IBM's strategy of going multi domain in MDM: all types of master data, whether it is Product or Customer data. Oracle Master Data Management SuiteOracle Customer Hub Customer data integration (CDI) solution. It provides following: A prebuilt extensible global master data model for customer information Services to maintain customer data across various systems and processes Capabilities to acquire, synchronize, and federate data Change and audit controls for data governance Oracle Product Hub To centralize all product information from heterogeneous systems. It provides:- Import workbench for improved data quality External transaction views. Mass maintenance capabilities to manage updates to multiple products Product sales configurations for manufacturing and service industries Change control capabilities Rich user interfaces for unstructured content management Predefined operational attributes with unlimited extensibility Oracle Site Hub It is a location mastering solution. It provides:- A prebuilt extensible data model (site master). Unlimited number of predefined and user-defined attributes for consolidating site-specific information Manages updates to multiple sites Site mapping and view using Google Maps® Prebuilt integration with Oracle Inventory, Oracle Property Manager, and Oracle Enterprise Asset Management. Web services to share data and integration with various systems. Hiperion data relationship management Previously known as Hyperion Master Data Management. To build consistency within master data assets. According to Oracle website "it provides the industry's first data model-agnostic master data management solution built to enable financial and analytical master data management in dynamic, fast-changing business environments". For more information please visit Oracle Website:- http://www.oracle.com/us/products/applications/master-data-management/index.htm SAP Master Data ManagementSAP NetWeaver is foundation for all SAP applications. It is SAP's SOA platform suite. NetWeaver is built upon open standards and inter-operate with, technologies such as Microsoft .NET, Sun Java EE, and IBM WebSphere. It can be extended by Microsoft .NET of Sun Java EE. Standard architecture for SAP MDM can have a centralized MDM server and clients connected it. Clients are connected to MDM server through SAP's Exchange Infrastructure using XML documents. Clients can also connect without Exchange Infrastructure. Implementation scenarios
Book - SAP NetWeaver for dummies By Jeffrey Word preview with limited chapters is available on google books. Sun MDM suiteThe Sun MDM Suite is based on the Sun Java Composite Application Platform Suite (Java CAPS). Sun MDM is developed in open source. It includes an infrastructure layer, Enterprise Service Bus (ESB) layer, MDM layer, and portal layer. The MDM layer includes the core MDM products. Following is description of these products:- Sun Master Index Defining reference data structure, storage and maintanance of this reference data. Sun Data Integrator For extracting legacy data from existing systems, transforming it and loading it into a master index database. Sun Data Mashup Engine Data aggregation from multiple sources to provide a single view of enterprise data. The Sun Data Quality and Load Tools Profile, cleanse, standardize, match, and load the reference data. It uses "Sun Data Integrator" for loading. Following are the tools related to data quality & loading:- Master Index Standardization Engine, Master Index Match Engine, Data Profiler, Data Cleanser, Initial Bulk Match and Load Tool I have summarized Sun MDM suite for the benefit of readers here but I have not described them in more detail because a very good documentation is availabe at Sun's site. For more information please refer:- http://developers.sun.com/docs/javacaps/products/sun-mdm-suite.jsp |
MDM (Master Data Management)
What is MDM ( Master Data Management )Master Data Management Management doesn't mean a technology all agree? Management is a practice, approach, anything but not technology. For the time being I am taking it as a practice. Master data means nothing but non transactional data. So what? Why am I worried about it. DBA's used to manage the database so master data too. Exactly. Than what are the challenges of managing master data? Now if an organization has various department, offices located at various parts of a country or even globe, in such a case even if all of them are part of a single organization and obviously they have data related to same organization but since they are located at different places and their functions are different they can have duplicate data, not matching with each other (in structure or definition). Inconsistencies may be there if master data which is common to whole organization is not defined and standardized and properly controlled, processed blah blah blah... So the challenges:- Consistency of organizations master data. There should be a process to do following things on master data of the organization:-
Obviously tools will also be required to do above mentioned activites. So MDM is a set of process and tools to manage master data of an organization. One important silver bullet- When we are talking about distributed department having their own copy of data, what does it indicate. Distributed environment requires silver bullet SOA :). So important to consider here is that MDM is very important for success of SOA. Impacts of poor integration of master dataCustomer satisfaction If you don't have a single view of a master / shared information there will be a communication issue. Say for an example in a mobile service provider company there is no standard shared master data of customer. If a customer has already discontinued the service, their billing department may continue sending monthly charges bills to the customer. This is my personal experience with more than one service providers. This leads to dissatisfaction in customers. This is not only applicable to a scenario of discontinued customer, this is also true for customer who have already paid their bills but still get calls from agencies who work for service provider. Operational efficiency What a provisioning department or billing department will do if it has different view of customer data. Obviously efficiency of the overall system will be impacted. Decision-support How can you trust a piece of information if you know what ever it is showing may not be correct. Something exactly same exist with different name of definition;. In such a condition how decision support system can work. Regulatory compliance Inconsistency may create issues related to regulatory compliance. You don't have a single definition of lots of things in the organization and hence you will produce wrong facts and figures in front of regulators. This can lead to compliance issues. Management of master data becomes more challenging in acquisitions and mergers scenarios where two organizations have their own data but ultimately it is going to be a single organization so some data must be common and shared. MDM (Master Data Management) Process
Following will run along with above activities in MDM journey. Data governance Body to set the rules, decide standards and monitor MDM process. Data stewardship Who will run the MDM projects for their own departments. Owners of data/information MDM - Approaches to maintain a Master ListOnly one copy of master— Single master data. Every change is directly made to the master data. Change in user of master data applications is required to use new data. Benefit of this is that you have consistency but it is not always practical. Many copies, one maintenance— Master data added to the single copy and changes sent to the systems where copies are stored locally. These applications can change the data which is not part of master. Benefit if this is that it requires minimal in the applications. Merging Master — Applications can change their their copy of the master data. Changes sent to the master where changes are merged into master. These changes are sent back to source systems to update local copies. Benefit of this approach is minimal changes in the source systems. There can be issues like:- Conflicts of same update by two systems, adding a new item in one system where other system already have that item will require remerge. Matching process to avoid duplicates in the master MDM - Styles to manage master listSingle Copy Master (Central Hub)- A single database for master data with all the information required for applications use it. After linking and matching data is consolidated at a central hub and published to various data sources. Central Hub prevents duplication. Currently running application may require changes to use master data. Many Copies one Look-up Service- Master data is maintained within user application databases. The central hub stores lists of keys. These keys are used to access attributes for a an entity and these attributes are linked to databases. To access master data, data is located, a query is distributed among various databases, and a list of the requested data is formed on the fly. This style doesn't require many changes in existing systems but in case of huge databases and addition of more and more databases this can become too complex for efficient access of data. Another issue I have already mentioned in my previous post is chances of duplicates in various systems. With addition of new databases queries may require changes. Mixed style- Mix of both the styles. Master data is kept on the native databases, keys and Ids are generated to access this data. Some of its important attributes are replicated to the central hub. Central hub can service the common requests, and queries are distributed among various native databases specific attributes (which are generally not frequently used). This improves efficiency of data access at the same time requires less changes in the applications. Update of replica may have issues in this style. Identification of attributes, standardizing format can be a challenging task. Role of Datanetworks, Datawarehouse & Datamart in MDMData networks Data networks facilitate in transmission of data within an organization. They provide information transfer and storage facility. A common server can have data which can be used by different departments or even group companies of a corporate. With data networks it is possible to gather master data from various sources. Obviously this will require consolidation and management. Another function data networks provide is sharing of important business information which is used daily transactions. Data warehouses Provide storage for the master data. Data can be analyzed and reported through it. Along with storage of data they also provide processes for the retrieval, analysis and management of data for proper access. Since they use common data model and they are optimized for data retrieval they facilitate easy analysis and reporting. This way source system (involved in transactions are relieved of this task and can work efficiently without degrading performance of source systems. Datawarehouses provide long storage of data which may not be possible in transactional db systems. Data Mart Although data marts can exist independently but often they are consolidated to dorm a datawarehouse. In some cases subsets of data is extracted to create data mart. Generally data marts are subset of datawarhouse. More specific data (related to a particular aspect) is kept in data mart. Individual departments or group of users use data marts to access data related to their business requirements. Since they are focused on specific information, frequently used data access from data marts is easy. Transactional Databases They store real time transactional data. Which MDM product is the best?MDM is becoming more important and relevant now a days for enterprises. There are tools, products which can help implementing Master Data Management but lots of vendors have launched MDM products in the market. Some are big players and some are small players but specialists. Which product to choose is a question which needs to be answered. MDM Selection Criteria Before choosing a product there are some criteria which needs to be prioritized for an organization and on the basis of these criteria a product supporting features in favor of the organizations requirements can be considered. Here are few things which can be considered:
To compare various MDM products I am conducting a survey through Survey Monkey to get feedback on these features to select a MDM product. I request you all to vote in this survey. It is a single page survey and doesnt require too much of effer. Just click and finish. Please click here to vote. This survey will provide enough ground for MDM product comparison. Which one is the best MDM? Answer to this question will depend upon what are the organzations priorities but these features will help them choosing a MDM product based on their priority. MDM readiness assessment toolIt asks about 37 questions on various pages. Questions are based on your general understanding of your organization to figure out what people in your organization think about master data, what do you do to clean duplicate data etc. Based on these and other types of question it sends you a report to your email ID. Important thing about this survey is it has questions touching various aspects of data:- Quality, Processing, Change Management, Organization, MDM Perception people have in your organization, Rules and Policies, Data Access & Navigation etc. Its report shows your score in various categories as compared to other organization which have undergone this survey. Since its score is showing your scores as compared to other organizations score it can be useful to know maturity of MDM at your organization. But at the same time it can be erroneous because organizations participated can not be a scale to know your maturity. |
SOA - SoaML
Last month "Revised joint submission SoaML for UPMS" was submitted by
some vendors to OMG. This was much awaited submission for me because I
wanted to know what finally they are putting on table. How they have
taken modeling for SOA is interesting to see. It describes a UML profile and metamodel for service oriented architecture's service design. Their aim is to fit it into model driven approach. It is based on the UML 2.0 metamodel L2. Important thing here is that they say it conforms to OASIS SOA Reference Model. Service is shown with "Pakage" notation. Every thing is covered but they have tried to accommodate everything using existing symbols and notations. This doesn't look natural for me. Specification can be found at http://www.omg.org/docs/ad/08-08-04.pdf I doubt this is going to make life simple. I doubt this will be used and if will be used will help people. I was unable to see any other aspect except old object oriented aspect in this specification. I believe this specification is created to facilitate existing tools so that no change in them will be required. What I think from SOA is that it is entirely different from normal OO paradigm. Architecture and modeling of it should be in such a way that can be understood by business people more. |
SOA design principles
There are some basic architectural rules and principles which should be
applied to a good architecture. All those rules , principles and best
practices are also applicable to SOA based system design but there are
best practices and principles which are specific to SOA because of
specific nature of SOA. SOA systems have some specific features due to
which design considerations should be special for SOA including basic
design principles applied to any architecture. Service Contract Service should implement an standard interface. This interface is a contract with service consumers. One of the goal of SOA is re-usability across organization and another goal is integration. In this scenario there will be various consumers of the service across and beyond organization. If service will not adhere to a contract, service consumers will not be able to use service. One of the most important design principle of SOA is conformance to service contract. Loose coupling Service contracts should be designed in such a way that they should promote loose coupling between provider system and consumer system. Since consumer and provider of service may be located at various places under control of different entities tight coupling will have ripples effects which can not be handled easily in case of any internal change in service provider. As a good design principle services should have coarse grained interface to improve loose coupling. Abstraction Another good SOA design principle is abstraction. A service contract should only have limited and essential information. Re-usability A service should be reusable on its own. Here this SOA design principle is considered due to commitment of SOA to promote re-uase. In general architecture rules or principles you might not have seen specific focus on reuse. General architecture design principles advocate about loose coupling, cohesion etc but they talk less about re-usability. SOA promises high ROI so reusability has special focus in good SOA design principles. There are two SOA design patterns help here - logic centralization and contract centralization. applying these two SOA design patterns results in to standardized and normalized service which fully support and maximizes re-usability. Autonomy This is one of the most controversial claimed tenets of SOA. As there was no unanimous definition of SOA and SOA world this SOA design principle (Autonomy) also has several meanings and understandings. I have serious objections and I will start a debate on it later but for the benefit of readers I will not make it an issues at this place and would like to summarize few things Bruce Johnson in his blog has discussed this controversial issue. He writes "A service, on the other hand, has a life outside of the calling application. If the calling application dies while the service is doing its think, the service will continue functioning properly. In fact, one of the guarantees of autonomy (in this sense) is the ability for a service to be called asynchronously with no expected return value. He further says "a service, where possible, should be self-healing". I support Bruce Johnson's views that it should not be called Autonomy. Autonomy is a wrong word chosen for expected behavior of a service mentioned by Don Box with respect to Autonomy. Autonomy means independence but what about scenario where a service will use other services to complete its task. A composite service. People have written that with loosely coupled, they are location neutral, services communicate through dynamically negotiated channels etc etc that is why services can be considered autonomous. I didn't understand why decoupling, standard contracts etc which are other design principles are used to establish this design principle (autonomy). Yes services should be designed keeping these things in mind (location neutrality, they should not know how other is working, they should have standard interface but how does it make a service autonomous? In practice no service can be autonomous. As Bruce has written "My own personal choice would be 'sovereign'". I still don't want to call it sovereign. Some explain service autonomy as "Service should have high level of control over its run-time environment.". This is funny? :) Statelessness There should not be long-lived relationships between consumer and provider of service. Any operation invocation should not rely implicitly on a previous invocation. Making service state full will have negative impact on availability and scalability of the services. Discover-ability Services should be easily identified and understood when opportunities for reuse are present. The service design should focus on “communications quality” of the service and its individual capabilities, regardless of whether a discovery mechanism for an example service registry is there or not. Composability Services should be capable of participating as effective composition members. Re-usability promise of SOA requires that new capabilities, business requirements can be fulfilled with no time by composing a new application using existing services. Good design effort is required to make sure it should promote composability. |
SOA testing
Due to specific nature of SOA eco-system it requires a new model
methodology and thinking for testing. Why? SOA services span across
boundaries of the organizations, applications are composite of various
services across the organization and beyond organization boundary. For a long time there was a need to have SOA testing model specially designed for SOA but it didn't exist. Here I am trying to put some thoughts on SOA testing. Challenges in SOA Testing
Testing Model Famous V Model comes into picture here. Test design analysis planning and execution happens throughout the life of SOA project. V model provides for parallel testing activities during project life cycle. Example- Requirement=Prepare UAT, Functional Specs=Create System Service Test, Tech Specs=Create Integration Test, Program Specs=Create Unit Test Why V model is suitable for SOA projects?
Requirement for SOA testing
Types of testing for SOA
|
SOA testing tools
Mercury Quick test, Service test, Load runner & Quality center. All of these together presented as a SOA testing solution. These cater for functional, regression test automation and performance testing. Acquisition of Systinet Corp has added SOA specific test capability in Mercury test suite. Parasoft SOAtest Automated webservices testing tool. WSDL validation, unit and functional testing, performance testing are main capabilities of this product. Addresses interoperability, security, scalability and change management issues of SOA. Green Hat GH Tester Tool for message based systems. Supports protocols like JMS, SOAP. Product supported by it include SONIC MQ, TIBCO RV. AdventNet QEngine Web-based test automation tool. Supported on multiple platform (built using Java). Caters functional, performance testing of web applications and web services. Borlad SilkPerformer SOA edition Tool for testing Webservices. It supports functional testing and performance testing. SOAP UI Open source desktop application which can be used for inspecting, invocation and developing webservices over HTTP. Functional, Load and Compliance testing are supported by this tool. Maturity of this tool needs to be taken care and evaluated as per specific organization SOA requirements. PushToTest TESTMAKER Open source framework to build test agents to check web applications for scalability, functionality and performance. Agents mimic user behavior. Mercury Quick test, Service test, Load runner & Quality center. All of these together presented as a SOA testing solution. These cater for functional, regression test automation and performance testing. Acquisition of Systinet Corp has added SOA specific test capability in Mercury test suite. Parasoft SOAtest Automated webservices testing tool. WSDL validation, unit and functional testing, performance testing are main capabilities of this product. Addresses interoperability, security, scalability and change management issues of SOA. Green Hat GH Tester Tool for message based systems. Supports protocols like JMS, SOAP. Product supported by it include SONIC MQ, TIBCO RV. AdventNet QEngine Web-based test automation tool. Supported on multiple platform (built using Java). Caters functional, performance testing of web applications and web services. Borlad SilkPerformer SOA edition Tool for testing Webservices. It supports functional testing and performance testing. SOAP UI Open source desktop application which can be used for inspecting, invocation and developing webservices over HTTP. Functional, Load and Compliance testing are supported by this tool. Maturity of this tool needs to be taken care and evaluated as per specific organization SOA requirements. PushToTest TESTMAKER Open source framework to build test agents to check web applications for scalability, functionality and performance. Agents mimic user behavior. |
SOA certifications
SOA- open source tools & products
JBoss Suite http://jboss.org/ JBoss has been one of the leader in open source. They have come up with JBoss ESB, jBPM (Business Process Modeling) (community as well as support subscription) additions. Other products of JBoss suite include - JBoss Rules, Hibernate, Seam etc JBoss Enterprise Middleware for SOA is enterprise edition (subscription based) See what is the different between community driven and JEMS: http://www.jboss.com/products/community-enterprise/ Community site: http://jboss.org/ Fuse Community http://fusesource.com/ ESB (based on ServiceMix) Message Broker (based on ActiveMQ) Services Framework (based on CXF) Mediation Router (based on Camel) Mule Source http://www.mulesource.org/ (community) http://www.mulesource.com/ (enterprise) ESB Mule Galaxy - SOA governance platform often reffered to as registry/repository WSO2 http://wso2.com/ Products are listed here: http://wso2.com/products/ Enterprise Service Bus Web Services Framework Data Services Server Web Services Application Server There are open source products offered by various communities :- OpenESB, Apache Camel, Apache Synapes, Blackbird ESB, ChainBuilder ESB, Jitterbit, Openadapter, PEtALS ESB, Openlink Virtuoso, ServiceMix |
SOA - websites and blogs
SOA product comparision resources
1-10 of 20