Oracle Enterprise Data Quality 12

With my new customer architect hat on, I’m looking across a broad range of applications and services to help my organisation be more effective and productive. As a large retail organisation, customer data quality is key to our ability to talk to our customers effectively. Imagine sending letters out to customers – in the digital age, that’s a pretty expensive channel but one which is required none-the-less. If we get the address wrong, or we have duplicate entries for a customer, we end up with a failed communication (really bad for our customer) or duplicate communications (annoying for our customer and bad for our bottom line). Oracle Enterprise Data Quality can help us resolve such problems.

Quickstats Profiler on the S_CONTACT table

I’m running Oracle Customer Hub (the Customer Master Data Management solution formerly known as UCM or Universal Customer Master), which is a Siebel vertical product that orchestrates and governs a process to take data in from sources, cleanse, enrich, match and de-duplicate, before publishing to consuming systems.

EDQ is a critical component of this solution, as it fills the cleanse, enrich and match capabilities of our end to end MDM process. It’s also an extremely impressive and feature rich product and I’m going to explore some of it’s features over the coming months. First things first, let’s download and install Oracle EDQ release 12:

  1. First up, you’ll need a Java JRE to run the front end components, so download and install a 64-bit JRE 8 from Oracle’s Java home
  2. Now, let’s get a trial copy of EDQ installed on a VM so we can have a bit of a mess around. Skipping the usual eDelivery route, head straight to the EDQ product page to download the product
  3. I’ve gone for the standalone 12.2.1.0 Windows installer version, since I’m running a Windows 2012 R2 server VM, but you can sit the newer 12.2.2.0 installation on top of a Weblogic server running on whatever host OS you choose
  4. Once downloaded, run the installer and simply following the instructions – it’s really straight forward for an Oracle product
  5. You’ll find a shortcut to the Launchpad – this is where you’ll find shortcuts to all of the features of the platform

    The EDQ Launchpad

  6. Launch Director, where you’ll set up EDQ to point to your data source and carry out activities. If you’re prompted to open a JNLP file, and your installation does not automatically associate this with the Java Web Start, then navigate to your Java installation folder and associate it with “javaws.exe”
  7. The default username and password is dnadmin/dnadmin – be sure to change the password to something memorable. Log in and you’re ready to go

    EDQ Director

Simple as that! Next time, we’ll connect EDQ up to our Siebel server database (you know, the one with all the lovely sample DB data in it) and run some profiling jobs.

 

So what is life like, after Siebel?

It has been many, many months since my last blog update. Life has changed considerably for me in 2016 with my two kids growing up, school drop offs, homework and being aeroplanes in the garden taking precedence over writing articles and installing Siebel patches.

In addition to my focus on family life, I’ve also closed down my consultancy (the imaginatively named “Ollerenshaw IT Ltd”), hung up my Siebel gloves and explored the world of full time roles in the great wide world.

It’s been fun, if a little daunting: I’ve worked almost exclusively with Siebel for over 15 years, so immersing myself in other technologies and roles has been difficult. I’ve found that my new roles involve a lot less hands on technology and I now find myself in the fuzzy world of architecture and strategy. It’s no bad thing, and I enjoy the challenge of taking greater responsibility for overall solution designs, working with businesses to understand what’s really important and how I can help them achieve those goals. To me, it’s an important role as it’s always been the case that shiny, snazzy Siebel solutions (including some of the “awesome” capabilities of OpenUI) often exist in the world to satiate the appetites of hungry developers – there’s often little to no business or user benefit in a lot of what happens in technically focused IT.

As part of my current role, I get involved in a lot of talk about strategy and planning for the future. It’s interesting to hear, from the coal face as it were, what’s getting the IT industry excited. I work for a retail organisation, and each vertical has different needs and wants and different solutions appeal, but for us there’s something that currently stands out: Big Data.

bigdata

Of course, we’re very excited here about the prospect of Cloud, including Infrastructure As A Service, Azure and AWS. We’re also excited about Software As A Service, which is now at a level of maturity where we believe we can actually start to map some of our business need onto these types of cloud services. Though not something we’re looking at here, Birst is a great example of SaaS: it’s Cloud BI by two of the guys responsible for OBIEE nee Siebel Analytics.

But Big Data is what it’s all about here. We’re talking Hadoop, MapR – we’re looking at some of the big players and some not so big. We’re also looking at data visualisation platforms – Tableau, PowerBI, Business Objects and many others. With a bit of luck, if the kids allow me the time to do so, I’ll pen some articles on my experiences in this ever growing buzz space.

Until then, wishing everyone in Siebel Land the very best for now and the future!

Installing Siebel 16 on AWS

Well, it was only a matter of time before I ended up working with Siebel again!

I’ve been introduced recently to Amazon Web Services – AWS to you and me. So this is what Cloud Computing is all about! I’m utterly gob-smacked by the power of this platform, the stuff you can do and the speed at which you can do it.

I was tasked with building a Siebel 16 installation using AWS and it is amazingly straight forward. Not to mention incredibly cost effective, compared to using physical hardware and on-premise virtualisation.

I thought I’d share with you the process of spinning Siebel up “in the Cloud”!

First up, my architecture involves two AWS instances – an RDS instance for the database and an EC2 instance for the Application and Web Servers. Using RDS, I’m able to spin up an Oracle Enterprise Edition 12c database in less than 10 minutes. It’s literally a few clicks and you’re done – no more slaving away with OUI, downloading files and the installation and configuration process. AWS captures parameters from simple web pages and does the rest – lightening fast. AWS will even keep my Oracle installation up to date, automatically applying minor patch updates during a maintenance windows that I can define.

SiebelDBAWS

So, I have a database instance up and running. I then spin up a new Windows 2012 R2 server. Again, the process is ludicrously quick. I choose from a number of types that have varying RAM, CPU and bandwidth allocations, allocate C and D drives with 60 and 120GB respectively and click a button. Less than 10 minutes later, I have an RDP file to download to my laptop that allows me to connect to the new server. I then jump on to Oracle eDelivery and download the Siebel installation media – I’m getting speeds of nearly 70MBs (that’s mega BYTES, not BITS!) and download the full Siebel installation in under 15 minutes. I unzip the media and backup the installation files to Amazon’s storage offering, S3.

SiebelE2App

I download and install a Java SDK, Oracle 12c 32-bit Client and SQL Developer then set up my TNSNAMES. I then run SNIC to extract everything and prepare my Siebel installation media and kick off my Siebel installation. Within another couple of hours I have a fully functioning Siebel environment.

While this is all pretty much “installing Siebel on a remote server”, the real benefit comes of now saving my Siebel Server installed as an Amazon Machine Image (AMI). This will allow me to VERY quickly spin up new Siebel 16 server environments for whatever I need – training, dev environments, the list is endless.

I’ve always been a bit of a cynic when it comes to “Cloud” computing but AWS has got me converted. This stuff is HUGE and it is AWESOME – with this stuff, server rooms are sure to become a thing of the past.

Siebel Repository Analyser – FREE!

After years of plugging away at my C# / Siebel experiment, I think it’s time to call it a day. I enjoyed working on the Repository Analyser mainly as an educational experiment for myself but also as a tool to help others. I’ve learned all I can from it and it’s time to move on to something else.

Siebel Repository Analyser C# Code

Siebel Repository Analyser C# Code

As such, from today the Repository Analyser source code is now available in a public Github project that you are free to use, modify and defile at your own risk.

I will no longer be enhancing the tool but will continue to support existing users via the forums.

Thanks to all who supported the development of the software.

The “EAI MSMQ Transport” Business Service

Inspire by the “reloaded” Siebel Hub Business Service Library, I felt motivated to write about the “EAI MSMQ Transport” business service. We use MSMQ extensively in my current project and I’m impressed with the simplicity and power of the technology. The JMS equivalent (“EAI JMS Transport”) has it’s own very nice article from @lex and works in much the same way as the MSMQ flavour. What follows is a short summary of some of the key features of the MSMQ technology, used alongside the various EAI functions provided by our favourite enterprise software.

The premise is relatively simple: Microsoft Message Queuing (hereafter refered to as MSMQ) allows systems and processes to communicate in a “fail safe” manner. That is, if the connection between systems were to break or disappear, message communication resumes when the connection is re-established. This is in contrast to socket or point to point communication mechanisms which require an “always on” connection.

Creating an MSMQ queue

Setting up MSMQ is simple, provided “Message Queueing” has been enabled on the Windows host. This is typically enabled via Server Manager on Windows Server hosts and through Windows Features on Windows Client hosts, such as Windows 7.

Creating a queue is equally as straight forward and can be done using PowerShell or via the UI. Via the UI, the process is easy:

  1. Right click ‘My Computer’ and select ‘Manage’
  2. Under ‘Services and Applications’ select ‘Message Queueing’
  3. Expand ‘Private Queues’ to see a list of queues enabled on that machine
  4. Right click ‘Private Queues’ and select ‘New > Private Queue’
  5. Give the queue a name and check the ‘transactional’ checkbox
    • A transactional queue allows blocks of messages to be treated as a single, atomic unit. If MSMQ fails to process that entire block, all messages that have been processed are rolled back
  6. Click ‘OK’ to create the queue

CreateQueue

MSMQ introduces the concept of Private, Public, Outgoing and System queues. We use Private queues in order to strictly control which service and administrative users have access to the queues. You can read more about the other queue types on MSDN.

Sending to an MSMQ

Sending XML to an MSMQ is simple, using the “EAI MSMQ Transport” business service. Two methods should peak your interest:

  • Send – this method will send a message to a specified queue. The method is “fire and forget” and Siebel will not be informed of the fate of the message
  • SendReceive – this method will send a message to a specified queue and await a response message. This allows for end to end processing, where the recipient sends either a confirmation of receipt or some sort of response or feedback from a process initiated by the outound message.

Method arguments that you’ll typically use are:

  • <Value> this should be initialised with the XML or text that you want to post to the queue
  • MsmqPhysicalQueueNamethe name of the queue to post to. This should be an MSMQ path such as “private$\mytestqueue”
  • MsmqQueueMachineName – the host name of the machine that contains the target queue. This can be localhost if you’re running everything locally

Using this Business Service method in a Workflow Process is usually combined with calls to the following EAI Business Services:

  • EAI Siebel Adapter.Query – this method creates a Integration Object instance from data in your Siebel Database.
  • EAI TransformationEngine.Execute – this BS method uses a Siebel Data Map to transform an IO instance into an external IO instance with a different schema
  • EAI XML Converter. IntObjHierToXMLDoc – this method tranforms the IO instance into an XML string, ready for transmission to MSMQ

An MSMQ outbound Workflow Process may look something like this:

MSMQSendWorkflow

Running the Workflow, with an appropriate “Object Id” of course, will result in an XML message containing your EAI query results on the queue. You can see it by going back to your MSMQ machine, expand your queue name and click on “Queue Messages”:

NewMessage

You could also invest a few pennies in the MSMQ Explore tool, available exclusively from Ollerenshaw IT Ltd!

Receiving from an MSMQ

Receiving a message is a little more involved, especially if you want Siebel to monito a queue and automatically process messages from it. Three additional elements are required to configure Siebel to systematically process messages from an MSMQ queue. There elements are:

  1. An “MSMQSubsys” profile configuration – this profile details the name and host of the MSMQ queue. The following parameters should be set:
    • MsmqPhysicalQueueName – the name of the queue to post to. This should be an MSMQ path such as “private$\mytestqueue”
    • MsmqQueueMachineName – the host name of the machine that contains the target queue. This can be localhost if you’re running everything locally
  2. A “EAITransportDataHandlingSubsys” profile configureation – this profile configuration specifies what Siebel should do with a message when it is received. Typical scenarios would invoke either a Workflow Process or a Business Service. The following parameters should be set:
    • DispatchWorkflowProcess – <Name of the Workflow process to invoke>
    • DispatchService – Workflow Process Manager
    • DispatchMethod – RunProcess
  3. A  “Enterprise Application Integration Receiver” Component Definition – this component effectively references the two profiles to “poll” the MSMQ and pass received message contents onto the handling Workflow or Business Service. Set the following parameters as a minimum:
    • ReceiverConnectionSubsystem – the name of the MSMQ Queue profile
    • ReceiverDataHandling Subsyst – the name of the Handler profile
    • ReceiverServiceName – EAI MSMQ Transport
    • ReceiverMethodName – ReceiveDispatch
    • Default Tasks – set this to 1 to ensure the component comes up and is listening from when the Siebel Server starts up

A Workflow Process, who’s name you’ll reference in the handler profile above, will look remarkabely similar to the Workflow you created for sending messages, using different methods of the same Business Services:

  • EAI XML Converter. XMLDocToIntObjHier – this method tranforms the XML message back to an Integration Object instance
  • EAI TransformationEngine.Execute – this BS method uses a Siebel Data Map to transform the external IO instance into an internal IO instance with a different schema
  • EAI Siebel Adapter.Upsert – this method applies your IO instance to the business layer, updating and creating data

A receiving Workflow Process may look something like this:

MSMQReceiveWorkflow

Using listeners and handlers is a mechanism to consistenly poll queues for messages, acting upon them as soon as they are received. It is possible to “manually” receive a message from the queue using one of the three “Receive” methods:

  1. Receive – simply receives / pops the next message from the queue and places it into a property
  2. ReceiveDispatch – receives the next message from the queue and dispatches it to a handling service
  3. ReceiveDispatchSend – receives the next message from the queue, dispatches it to a handling service then sends a response message

MSMQ, and other queuing technologies such as JMS, are a simple and elegant mechanism for providing failsafe, transactional communication between systems. The Siebel EAI business service family provide everything you need, and more, to extract, manipulate and manage structured message data using industry standards. Combined, you have a very easy to use mechanism for Siebel to communicate with the outside world.

 

The Siebel Documentation Game

I’ve worked with Siebel for a very, very long time. As such, I’ve encountered many different design and documentation paradigms – including Oracles own OUM paradigm.

How to design a Siebel system?

For many years I’ve encountered what I consider to be a dreadful practice – the “Low Level Design” or “Technical Design” that’s essentially an export of the Siebel repository. I’ve seen lengthy documents filled with tables showing field definitions, BC user property settings, eScript code, you name it. I’ve also become aware of a tool that will generate Word tables using a connection to the repository. Tsk!

NoTables

Bad!

My suggestion is this: such documentation serves no purpose and offers little value – in fact, it’s counter productive.

Let me elaborate. Typically, a customer starts off with a set of requirements. These are then translated to system functional requirements and incorporated into a functional design. Then comes the technical implementation artifact.

I believe that a technical design document should provide the following:

  1. Traceability to the functional design, showing an encompassing system specific solution that meets the functional requirements
  2. A demonstration that the technical design approach is consistent with the technical and architectural best practice of Siebel and your organisation
  3. A source for a technically adept, proficient Siebel professional to implement an appropriate solution
  4. A source for unit test scripts

What it should not be is a load of tables showing Tools configuration steps.

Why? I even have a tool to export to Word!

Put down your tool and walk slowly away. Do not approach the tool – it will be safely disposed of. Using fire.

To me, good Siebel development produces self documenting code and configuration. By using comments field and incorporating comments into scripts, you can pretty much explain what you’ve done and why at the repository level. Why would you add this to a design document? Who are you trying to impress? A design is about forward planning and is a tool for communicating with your team and other assets or systems that may be impacted. You should not be creating / defining Tools objects at this stage. Some points to consider:

  1. Tools configuration is build – not design. Design should focus on the approach for meeting requirements and demonstrating consistent best practice
  2. Configuration changes – Siebel provides many ways to skin a proverbial cat. At built time, the developer will select the appropriate mechanism to implement a functional requirement. Why ham string yourself at design time? Decide on the right approach at the right time.
  3. Maintenance is a nightmare – reverse engineering every change made to a repository into a document is time consuming, annoying and unlikely to happen! How many developers like documentation?

I’ve always recommended a “light touch” approach to technical design documentation, demonstrating an understanding of requirements and implementation best practice over a dump of obviously already checked in code and config. Anyway, I can always rely on my highly paid Siebel developers to do the dirty work and put it all together. That’s what they’re paid for, after all.