Slow Performance On Node -- Root Filesystem maybe 100%

When SSH over to one of the nodes in our RAC cluster the response was horrible, in the neighborhood of 3 minutes plus before being prompted for username and then another 3 minutes plus to be prompted for password.

First I pull up TOP. The first thing I noticed is the Run Queue

load average: 23.09,22.48,21.73 -- Ouch

I check the CPU:

Cpu(s): 0.3%us, 0.6%sy, 0.0% ni, 74.4%id, 24.7% wa, 0.0%hi, 0.0%si

So it's not a process in the CPU, however, I have a high wait at 24.7%

I check the MEM:

Mem: 65742800k total, 22487544k used, 43255256k free, 680408k buffers

That is definitely not right as I should have around 57700140k used. This looks like the instance may not be up.

I check the instance:

ps -ef|grep pmon

and only the asm1_pmon is running

I check the filesystem space

df -k

And the root filesystem shows / 100%

That's the answer. I'm using ASM, root filling up can cause the instance to go belly up. It will also cause the run queue to jump as processing that need the root filesystem will not be able to process.

The resolution is to find the largest files in the root filesystem in remove them. In our case the trace file cleanup routine is not functioning properly and I find trace files dating back 2009 to include the cdmp and core dumps. I remove all the trace files. I also zip a few large files that have appeared recently that I know are not being actively utilized and will look to remove them.

I try to restart the instance

srvctl start instance -i orl1 -d oral

But receive a dependency error on the resource, so I opted to bounce the node which will also restart the CRS and both instances.

If your CPU is low, but your run queue is high, check the root filesystem. Its the most common issue we see in this scenario.

ASM Issues When Adding A New Node

It seems with every new node that is added a different issue slaps us in the face.  It would be so nice if just one node went in smoothly and without incident... of course it would be even better if resources were available to test adding the nodes, but that's not the issues I want to document in this post.

There have been a few issues with ASM during this exercise of adding nodes; ASM Instance Number incorrect, and Disk partitions not shared across all nodes -- actually has happened twice but for 2 different reasons.

Issue 1:  ASM Instance Number Incorrect.  The ASM instances are using an spfile

$ORACLE_HOME/dbs/spfile+ASMx.ora

The x is replaced with the number of the ASM instance.

Once the configuration is set in DBCA, the utility asks this is an ASM database would you like to extend the ASM instance?  Of course, the answer is yes or there will be problems when the database instance is extended -- Disks will not be available.

However, this bombs with the Instance Number is incorrect.  It seems that the spfile that DBCA decides to use is a generic version

$ORACLE_HOME/dbs/spfile+ASM.ora

That's an spfile without the instance number attached.  Since you can't edit the spfile directly, back on the original install host create a pfile from the spfile while logged into the ASM instance

create pfile from spfile;

I'm good with the location, but if you want to specify the directory and filename that can be done with

create pfile='/oracle/products/dbs/init+ASM1.ora' from spfile;

Now edit the pfile, I add the new instance number as well as any instance numbers that maybe missing.  Then recreate the spfile file.

create spfile from pfile;

Then I copy this file over to the new nodes spfile+ASM.ora generic file.

Rerun DBCA and the ASM instance is extended, as does the database.

Issue 2a:  Disk Partitions not shared across all nodes

The first time the message was received only one of the partitions wasn't being seen.  The fix:
1.  Ensuring that Oracle was in the Disk group (sets the permissions to access the disks)
2.  Check the raw devices in the /raw/dev file to ensure they are correctly specified and not duplicated.  This
     will be on the new node.

And in case anyone is wondering how I did 1 and 2, I'll have to defer to my boss as he's actually the one that handles the raw devices and ensuring the powerpath ids all match up.

Issue 2b:  Disk Partitions not shared across all nodes and all of the diskgroups are listed

Metalink Note: describes this as symbolic link is used on the instance the initial install was performed from, and the spfile needs to be re-created without the symbolic link and the ASM instance bounced.

Since this is my fourth node and this is the only one I have had this issue with I'm inclined to believe it has something to do with how I created the spfile and the pfile that points to the spfile.  So tomorrow I get to experiment on how to correct this situation.  My goal is to correct it, without having to bounce any of the ASM instances.

So tomorrow should be an interesting day.

I still don't have a solution to this problem. I did open a SR with Oracle which they haven't provided any useful information to try only asked for information that was already provided and files that were already uploaded. I often wonder if the support analyst read the actual information or just make assumptions so that they can push the issue back to the customer and get the clock off of them.

Adding A Node To A RAC Environment Gotcha

Over the last month I have had the opportunity to add 4 additional nodes to a RAC environment.  The current environment was 4 nodes running Suse 9.  The decision was made that instead of replacing the existing nodes, we would add and make it an 8 node cluster.  If you had the opportunity to read my post on CPU Utilization, you will understand that what we really did was increase our capacity by increasing the availability of CPU time for more transactions.  Our Arrival Rate can increase without impacting our response time -- provided of course we can balance our workload across the nodes appropriately.   On the horizon is an upgrade to 11gR2, so the new nodes are running Suse 10.  The older nodes will be removed as we have the new one's stable and upgraded to Suse10.

The decision to run with a mix OS for a short period of time was not without at least one quirk.  If you have an OCFS2 mount point you will not be able to mount it on both versions, its either 9 or 10.  This is unfortunate even in the short term.  The use of external tables or even jobs that still use UTL_FILE will have fail if they happen to start on one of the nodes the filesystem is not mounted on.   A work around would be service groups, however that may require some code changes that might not make sense in the short term.

A work around that has been effective in the short term, is to set the local_listener on the nodes that do not have the OCFS2 filesystem to a listener where the filesystem is mounted.  Let's say I have a host ORL1 and ORL2 that have the filesystem mounted, but I have ORL3 where the filesystem can not be mounted at this time.

Ensure that the TNSNAMES.ORA on ORL3 has the Listener for one of the two nodes where the filesystem is mounted:

LISTENER_ORL1 =
  (address = (protocol = tcp)(host = oral.medai.com)(port = 1521))


Then I change the parameter local_listener to one of the 2 hosts listeners:

alter system set local_listener=LISTENER_ORL1 scope=memory sid='orl3';

This changes the parameter only in memory and only for instance ORL3.

Now if any session connects to the listener on ORL3 it will be redirected to ORL1.  Of course, this means no jobs will be directed to ORL3 which defeats the purpose of adding the additional nodes.  This is only a temporary solution to adding the new nodes before removing the older nodes for upgrade of the OS.

Manipulating the local_listener parameter is also a way to redirect jobs away from a node that you may be preparing to perform maintenance.  This allows jobs to continue to process that have already started, but will prevent additional jobs from starting on the instance.  Allows you to bounce the instance, or node in order to perform maintenance without having to kill processes.  Doesn't solve the issue if you are using inter-instance parallelism and you want to take down a instance or node for maintenance as that is a whole other discussion.

Although its not supported and Oracle strongly suggests not running in a mix OS version, it does work and will allow you the opportunity to upgrade your OS one host at a time.   But its not without quirks and you need to ready to either handle the quirks or research the quirks.  The OCFS2 is just quirk, I'm sure there are more I just haven't been able to pinpoint the actual issues we have experienced with the different OS versions.  Although I will say each node that I have add different problems have appeared, but I think that's just Oracle and the utilities they have created to make things easier.

If anyone has some other quirks or gotchas please feel to list them out.