Critical issue with upgrade to IP2014: SBL-SCL-00143

UPDATE: As @lex points out, this is hinted at in the Installation Guide and referred to in the Siebel Security Guide:

NOTE:  When you upgrade to the current release, the Siebel Server system service password, which is required to connect the Siebel Server to the Gateway Name Server, is automatically reencrypted using AES encryption. The Gateway Name Server Password parameter, which is set at the Siebel Enterprise level, is also automatically reencrypted. You do not have to reencrypt these passwords manually.

Shame the installation doesn’t seem to do either! 🙂

I reported last week that I had significant issues with the upgrade to IP2014: I ended up with an “un-startable” Siebel Server and yet another open Service Request (still unanswered, I might add).

One of my readers and valuable contributors, Lance, has done some investigation and has found the solution to this problem: a new encoding mechanism (AES instead of RC4) for the password component of the Windows Service / Linux daemon command line and SIEBNS.DAT has been implemented and is not updated correctly by the installation process.

I’m not even going to begin to rant about how disappointing this is in terms of Oracle’s failure to test their software – it’s a story I’m really quite tired of repeating. So, on with the solution.

Recreate the Siebel Server Service

Execute the following on the command line:

This should update the ImagePath with the newly encoded password, allowing the Siebel Service to start.

Fix SIEBNS.DAT

Open a copy of your SIEBNS.DAT file in Notepad. Note down the encoded enterprise level Password parameter.

Run the following command line:

This should fix SIEBNS.DAT and allow all of your components to start.

8.1.1.14 Patchset 1 – First Impressions

Well, after my miserable attempt at upgrading 8.1.1.11 to 8.1.1.14 (which has left me with a completely unusable Siebel Server – more on that in a future post), I’ve downloaded and applied 8.1.1.14.1 – aka 8.1.1.14 PS1.

Download and creation of the installer is much the same as always, no dramas. SNIC seemed to take a bit longer than usual but I still ended up with the usual installation folders.

Successful? Really?

Success!

PS1 is an “opatch” installation and, though I used to hate opatch, it has grown on me. It’s actually really straight forward to apply after setting ORACLE_HOME and JAVA_HOME. Something deep inside me yearns for a UI front end to this though – too many years of being spoilt with InstallSheild.

Installation took a while and it looks like every file and binary has been shipped, not just those updated, which means pretty much everything is replaced. Hopefully that will cure my earlier SBL error but I have a gowning fear that this is because I’m still running 32-bit Windows. Oracle have always stated that 32-bit OS’s would soon be removed from support and it looks like the time has come. As I understand it, the Siebel dev team’s build environment has also moved to a 64-bit platform from 8.1.1.14 and onwards, so it would all make sense.

There’s a new version of the ActiveX control, which is a real pain for us. We just want rid of it altogether and it’s a real bind to deploy to our locked down client estate. Hopefully we will have fully deployed to OpenUI on 8.1.1.9 before we get to applying 8.1.1.14PS1.

As others have noticed, PS1 completely buggers up your CFG files – nice bit of testing there! This is easily remedied by copying the files back from the backup folder created by the original 8.1.1.14 installation process.

Having applied my PS1 updates, I’m still unable to start up the Siebel Server Service – same error as before. 🙁

Back to the drawing board and a Windows 2012 64-bit server, methinks! 🙂

8.1.1.14 (Innovation Pack 2014) – Further Thoughts and Bugs

UPDATE: I have now completed the upgphys and compiled my SRF. On starting the Siebel Server Service the whole thing is crashing with the following error:

SBL-SCL-00143: Internal: input disassembly failed

Absolutely no idea what this means but I think it’s time to give up on this and get on with applying PS1!

Having given up and skipped the “Generate Merge Report” step in upgrep, the rest of that process completed without error. I ended up with 4 Repositories and a job lot of merge conflicts to go through.

As this is a PoC, I decided to skip the conflicts for now and moved straight on to the upgphys, encountering my first bug straight away: upgphys allowed me to progress WITHOUT running the Siebel Tools command line and setting the “Conflict resolution completed” checkbox. According to Bookshelf, this should not be possible. The may be due to an earlier upgrade from 8.1.1.9 to 8.1.1.11 but it looks like it’s a bug, regardless.

upgphysfail

Upgphys Fail

I then encountered more problems with some of the new vanilla objects introduced in 8.1.1.14. In particular, we had defined a custom index on S_AUDIT_ITEM that the IRM had “duplicated” as a new index with the same columns. Though the DDLSync process detected this instance and tried to fix it, by removing the original, it still resulted in a failure of the upgphys process. Probably not a bug, as I should have identified and resolved the repository conflict prior to running this part of the process.

Having corrected this (I had to manually update the DDL to allow it to continue) I hit another potential bug: part of the new process renames two of the repositories as “v8_1_1_11 New Customer Repository” and “b8_1_1_11 New Siebel Repository”. Now, I’m not sure what 8.1.1.11 is doing in there (apart from being the baseline version from which I upgraded) but the processes then attempted to run an exprep (“Exports repository to a file”) using the command:

This is obviously expecting a different Repository name and so it fails with the rather cryptic error: “Dictionary is not available on this database”

Now, I’m not sure if this is because of something I’ve done wrong or because I’ve had to restart the process so many times to get it to work, but I’m not sure how to continue. If this is a bug, and the repimexp command line is simply wrong, then I’m in trouble. If it’s right, and something I’ve done has resulted in the Repositories being incorrectly renamed, then I can resolve this in Siebel Tools. As it is, I think I might call it a day and roll back my whole environment.

My deduction from the exercise so far: wait for Patch Set 1 before doing anything upgrade related with 8.1.1.14.

8.1.1.14 (Innovation Pack 2014) – First Impressions

UPDATE 28-Nov-14: Word from Oracle is that the “Merge Report” issues have been addressed in PS1. I’ll update further once that’s released and I’ve had a chance to test it.

I’ve finally had time to download the IP 2014 / 8.1.1.14 installation media and have a crack at applying it to my 8.1.1.11 Proof Of Concept (POC) environment.

Installation of the binaries was really straight forward and the new OUI installers really have improved since the old days of 8.1.1.8 and below. I had no problems whatsoever patching client and server components and the process was quick. I think Oracle may have tweaked the “backup” process that plagued 8.1.1.9 to 8.1.1.11, so all went well with this stage.

UpgradeFailed

Wizard Fail

Executing IRM is once again as straight forward as running the Database Configuration wizard and selecting “Upgrade Database”. Don’t forget to download and install the Ancestor Repositories first though. I still encountered the same issue as 8.1.1.11 in that Siebel Tools prompted me to select a Repository when IRM had finished importing the 3 new Repositories. Oracle need to fix this as it makes an unsupervised installation impossible.

The merge itself took less than two hours, which was most unexpected. Oracle have stated that the volume of repository changes between 8.1.1.11 and 8.1.1.14 was far lower than with previous IRM’s but I was pleased with the performance of the merge.

However, I’ve been unable to continue the process past the new “Merge Report” step. This is failing for a number of (cryptic) reasons that required me to look into the MergeReport.jar file shipped in the Database Server “COMMON” folder:

  1. The process does some checks against your Windows OBDC settings and it does this using a hard coded repository path: HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\ODBC\ODBC.INI\
    • If you are running Siebel on a 32-bit Windows Operating system, this will fail. I got around this by exporting my ODBC.INI registry settings, modifying the file and re importing. This is not supported, though, so don’t go trying this at home!
  2. The process creates some tables (S_MERGE_ALLDATA, for example) in the SIEBEL schema, but hard codes the tablespace name
    • The process is hard coded to attempt to create these in the “DATA” tablespace of the Siebel user schema. In my environment, I call my table space SIEBEL_TAB so the process fails. The wizard accepts data and index table spaces as input so there’s no excuse for these to be hard coded. I’m currently looking for where I need to hack the correct values to get the process to complete but worked around this for now by doing a temporary “ALTER TABLESPACE RENAME TO”
  3. The process now fails with an error: “Error in SplitFile function1”
    • I have no idea what this means or how to resolve it and I’m giving up! This new piece of functionality clearly hasn’t been tested properly and I’m not wasting another minute on it! 😉

So, another unfortunate experience with the process. I might just skip the Merge Report for now and let the upgrade finish so I can get on with the conflict resolution and compilation, but it’s frustrating to have to “bypass” one of the touted new features. If and when I find a solution, I’ll be sure to post.

Stay tuned for a first look at the new functionality and look and feel of the new UI and responsive theme – coming soon!

Siebel Upgrades and the 64-bit OS

I mentioned a while back that Oracle seem to be moving their supported platform matrix towards purely 64-bit Operating Systems. From 8.1.1.11 and 8.2.2.4 onwards, Siebel software is only supported on 64-bit OS’s.

With the new features of Siebel IP 2013, upgrading is becoming a far more attractive proposition than before. If you’ve ever been involved in putting together a business case for a Siebel upgrade, you’ll understand what I mean.

Now, with a number of existing 7.8 and 8.1 / 8.2 implementations sitting happily on 32-bit architectures, we have a genuinely intriguing challenge when it comes to upgrading to 8.1.1.11 or 8.2.2.4: what do we do with the existing infrastructure?

Clearly, there are options to instigate a “technology refresh” and replace ageing servers with shiny new ones with 64-bit OS’s. Equally, there will be organisations who have invested greatly in their kit and do not want to have to re-buy their infrastructure to support a point release upgrade to their enterprise software. Looking at Windows specifically, there is no upgrade path from 32-bit to 64-bit operating systems. Some organisations will be lucky enough to have virtualised infrastructure that can easily be “swapped” out, but there are plenty of organisations still running large Production enterprises on physical boxes.

So, I’m really reaching out to the community to ask: what would you do?

Answers on a postcard or via the comments section below!