---

# Interrupt tuning: Search for Wasteful Hardware Components

For enterprise servers, do not be surprised if Processor: % Interrupt Time is around five to twenty percent of the total CPU workload, particularly if the network and disk I/O is very heavy. There are, however, some components that behave better than others. Attempt to obtain NICs that truly support bus mastering and disk host bus adapters that support DMA transfers versus PIO. Some disk and network adapters operate more efficiently than others, thus they require less CPU cycles to operate. Trade magazines provide numerous comparisons of these products. One good source of information is the NT Magazine web site located at: http://www.ntmag.com.

Remove Faulty Hardware Components

When a hardware device, such as a NIC, interrupts the processor, NT Server's Interrupt Handler will execute to handle the condition, usually by signaling I/O completion and possibly issuing another pending I/O request. Observe the Perfmon counter Processor: Interrupts/sec. If this number begins to grow compared to your baseline when under normal workload, there is a good possibility that a network device has become faulty. When a device becomes faulty, it may begin generating high numbers of interrupts, which inundates the CPU. This wastes precious CPU cycles. Replacing the faulty device will alleviate this situation.

Processor : % Interrupt Time. This is the percentage of time that the processor is spending on handling Interrupts. Generally, if this value exceeds 50% of the processor time you may have a hardware issue. Some components on the computer can force this issue and not really be a problem. For example a programmable I/O card like an old disk controller card, can take up to 40% of the CPU time. A NIC on a busy IIS server can likewise generate a large percentage of processor activity.

---

However, there are no indications of errors in the system log on the server. The server generating the alerts is a backup server which will be intense both on network and disk I/O so it matches the condition defined above. The server is not a virtual so existing tuning information does not apply.

To determine what the interrupt time was at, the simple way is to open the monitor generating the alert and see what numbers appear each time the state changes to critical. From our server, the general average was less than 12. 

The product knowledge provides a view that indicates that we can see this performance counter trended over time but there is no data when the “Start Processor % Interrupt Time Performance View” is executed.

This performance view has no information because the rule which gathers the information is disabled by default. While that makes sense in general (why store something in the Operations Manager database and the Data Warehouse unless you need the information) we wanted this information so we could better determine what the actual trend looks like for this server.

To enable this counter we created an override to enable to rule to collect the performance information: (we also changed the tolerance sample from 5 to 1 because we were not seeing data in the graph but this is not shown in the graphic below).

Next we trended this server over time to determine an actual regular baseline for the server. If the server is functioning normally during this timeframe we can now change this setting to the higher threshold.

After watching this counter for a longer duration we got a better picture of what were normal levels for this counter on this server.

As a results of determining the trend for the server (and verifying with the server owners that the server was performing as expected during the period of time we were monitoring this counter) we created an override for the server to 20 based upon the performance history for this counter on this server (shown below).

Once we had this override in place, we removed the original override for the rule which gathered the counter.

Summary: The Total Percentage Interrupt Time monitor can be tuned on a per-server basis by determining what the trend is for the performance counter during a healthy period of time and then creating an override for the server to the new threshold.

I'm trying to set up an alert-generating rule to generate alerts for specific syslog messsages, and am running into a bit of a wall. Perhaps someone else has gotten this working?

As a bit of background, I've configured one of my monitored Cisco devices (an ASA) to forward syslog messages to my RMS. I also have the xSNMP Syslog Alerts working, so I know that the RMS is, in fact, receiving syslog messages. (A nestat -an | findstr 514 also shows that the RMS is listening on UDP 514)

However, while the xSNMP Syslog Alerts look for syslog events by severity, I would like to search for specific messages, by looking for certain event #'s within the "message" parameter.

While I want to look for specific events, such as 733104 & 733105 (Possible Syn Flood attack), for this example/as a proof-of-concept I am looking for 106001, as those are presently being logged. In other words, when it's working, I'll know that it's working within a few minutes.

The goal is to be able to set up an Alert-Generating Syslog rule raising an alert for syslog messages where "facility" = 16, and "message" contains "106001".

A sample alert (from xSNMP alerts) looks like:

Facility: 16
Severity: 2
Priority: local0.critical
Timestamp: Sep 3 20:28:16

Message: Sep 03 2010 14:29:22 FW1 : %ASA-2-106001: Inbound TCP connection denied from 69.69.31.28/1199 to xxx.xxx.xxx.xxx/135 flags SYN on interface Internet

I've followed the instructions at both, with no luck:

http://contoso.se/blog/?p=158

and

http://support.microsoft.com/kb/942863/en-us

...which slightly differ (the first one has the rule targeted at the MS, the second has it targeted at an agent, if memory serves). I've also tried rules targeted at "xSNMP  Device" and "SNMP Network Device" to no avail.

Here's the rest of the steps I have taken in my rule attempts:

1. Configure my Cisco ASA to forward syslog messages to my RMS.
2. Authoring > Rules > New Rule
3. Rule Type = Alert Generating > Syslog (Alert)
4. Entered rule name & description
5. Rule category = Alert
6. Rule target (I've tried Root management server, agent, SNMP network device, xsnmp network device; should I be trying something else?)
7. Expression: "facility equals 16" (I decided to keep it simple and try to get everything, and then attempt to filter it down by a string with the messsage parameter; better more results than none, to begin with)
8. Set up the alert name, description, set severity to "Information" (for now, so any output doesn't get lost in the xSNMP alerts, which are all Critical & Warning).

The properties/configuration of the completed rule(s) shows the Data Source as "SyslogDS" and the Response as "Alert".

So far, no luck.

So, I guess the question is, how does one create an alert-generating syslog rule (that actually generates alerts)? I think the problem lies either within the targeting of the rule, or maybe the rule category?

Any insight appreciated!

Thanks!

I found a blog somewhere (wish I had the link to give them credit) that said that sometimes even though you use an elevated cmd prompt it doesn't always work.  It said to disable UAC, reboot, and try again.

As soon as I did that, everything worked perfectly.  No idea why, but it did.

Our RMS is on windows 2008 Enterprise sp2 64 bit.

Our issue is that we start up the Exchange Correlation Engine and it will eventually cause the MonitoringtHost.exe process to consume all memory.  We know this stems from the Correlation engine.  we are working with MS on this but we have a counter part with the same management pack but on 2003 for their RMS and this does not happen to them.

We have not disabled any exchange 2010 rules which is a "known" issue if you use the 2010 MP.

anyone seeing this?

As suggest, I try to run the script manually using the command line. The script complete sucessfully and return the expected XML.

The Mp containing the vbs is available here : http://pastebin.com/JL9KtQJm

Using the function LogScriptEvent(), I manage to figure out the call to oAPI.Return() hang when the propertybag is too big.

OpsMgr 2007 R2 CU2 rollup hotfix ships – and my experience ...

SCOM really amazes me sometimes, and sometimes it simply sucks... :-)

Figured out i'll go for optin one short term, and make a option 2, longer term.

I started creating MP in Auth Console. -> Probe Action -> Picked Microsoft.Windows.Script.PropertyBagProbe module put my script in etc.. but on save it says ... this ... can you help :-) ?

error. ProbeAction Module (Type=Microsoft.Windows.ScriptPropertyBagProbe,  ID=Script) cannot be a leaf node in this composition. The composition has no input types (but this ProbeAction leaf module has one [ManagementPackElement=System.BaseData in ManagementPack:[Name=System.Library, KeyToken=31bf3856ad364e35, Version=6.1.7221.0]]). Only P....

Cheers,