Smarts Dynamic Model: Instrumentation voodoo

In the previous couple of posts about Smarts dynamic model, I've added a new class, created new properties and defined events using Smarts MODEL language. Our use case has been monitoring of applications and I've demonstrated how dynamic model can be used to easily add the necessary functionality.

I had not yet worked on how to instrument the object properties (so far used dmctl commands to simulate property value changes). The main instrumentation method in Smarts is SNMP. If there is an SNMP agent that reports the status of applications we'd like to monitor, we can use the SNMP instrumentation mechanism to poll the agent to monitor the status of the applications.

Unfortunately, at this point, we are leaving the land of the "easy" and move into the realm of black magic. Using SNMP instrumentation via dynamic model is not straight forward. There are very few examples, and limited documentation, and when something does not work, it is quite difficult to figure out why for mere mortals like myself. I find it quite difficult to troubleshoot problems etc. Most of the difficulties arise from lack of information, examples, etc. more than shortcomings of the technology.

SNMP Instrumentation class

A separate class of objects are used for SNMP instrumentation. The model code below creates a new class to poll a specific part of an SNMP mib.

interface ApplicationProcessInstrument:ICIM_Instrumentation
{
instrument SNMP {
ApplicationProcessStatus = "1.3.6.1.2.1.4.1"
};

instrumented attribute int ApplicationProcessStatus
"Is application process up or not 1 is up 2 is down";
}

The code segment above specifies an SNMP OID to be polled and assigns the value of the OID to an attribute called ApplicationProcessStatus. Note that the OID above has nothing to do with application processes. I don't currently have an SNMP agent with host resources MIB, so I'm using a bogus OID.

There are two gotchas here:

1. Remove the dot in the beginning of the OID (as above)
2. Beware that there is an index at the end of the OID that is not here.

So if you walk the MIB, you'd see the OID as .1.3.6.1.2.1.4.1.0 and not 1.3.6.1.2.1.4.1

Propagation

I had used the propagate command before as explained in the previous post. I need to use it here as well, to transfer the ApplicationProcessStatus property from the instrumentation class to the application class. So the MyApplication class I used in previous posts would look like the following:

interface MyApplication : ICIM_LogicalElement
"Instrumentation class used to monitor status of a database."
{
relationship HostedBy, Host, Hosts;

propagate attribute int min MyAppStatus
"is the application is running or not?"
<= ApplicationProcessInstrument, InstrumentedBy, ApplicationProcessStatus;

event Down "The application seems to be down"
= MyAppStatus == 2;
export Down;
}

MyAppStatus property of the MyApplication class object will have the value of the ApplicationProcessStatus property from the ApplicationProcessInstrument class. If the value is 2, then it'll generate an event which will be received by SAM and a notification will be created in SAM stating that the MyApplication object is down.

We will also keep the relationship between the Host class and the MyApplication class, as we will need it later on to figure out which SNMP agent to poll.

refine interface Host
{
relationshipset Hosts, MyApplication, HostedBy;
}

I now have the model I need to poll an SNMP agent to populate the values of object properties. Easy right? well, not really. The problem is working with tables in the MIB is complicated. It is also not so easy to figure out which entry in the table corresponds to which application process, etc. Frankly, I have no idea how to do this. I can only hope that the gurus in EMC may provide some examples where an SNMP table is polled for instrumentation so that we can all be enlightened.

Moving on, we now have the model, yet our work is not complete. We need to create the relationships between the instrumentation and application objects. There is an asl script provided with the dynamic model tutorial example 3, load_accessor.asl. I've used this script as a template and modified it to work with my model. Main modifications are:

1. it works with the MyApplication object instead of the Host object
2. it traverses the HostedBy relationship to find the SNMPAddress property of the Host (application is running on)

It looks like this:

Debug = TRUE;

default Timeout = "700" ;
default Retries = "3";

default appName = "MyApp1";
default InstrumentationName="ProcessInstrument"; // this variable can be passed in via the sm_adapter -D option

// SNMP Polling Parameters

default ReadCommunity = "public"; // this will be ignored for SNMP v3
default WriteCommunity = "public"; // this will be ignored for SNMP v3
default AgentVersion = "V1"; // V3 Specific Polling Parameters
default EngineID = ""; // Can be left blank
default EngineBoots = 0; // Leave as 0
default EngineBootTime = 0; // Leave as 0
default UserName = ""; // If authentication enabled, set to a valid user name
default AuthProtocol=""; //To enable authentication, set to "MD5" or "SHA"
default AuthPassword = ""; //If authentication enabled, set to the user's password
default Context = ""; // Can be left blank
default PollingInterval =30;
default index="";

poller = object(getInstances("SNMP_AccessorInterface")[0]);

START() {
eol
}
do
{
// Convert the time to milli-seconds
Timeout = numeric(Timeout) / 1000;

appObject = object("MyApplication",appName);
print("Application object name is ".appObject->Name);
if (!appObject->isNull())
{
print("app is hosted by ".appObject->HostedBy);
hostObject = object(appObject->HostedBy);
snmpAddress = hostObject->SNMPAddress;
print("SNMPAddress is : ".snmpAddress);
applicationInstrumentation = appObject->makeInstrumentation(InstrumentationName);
print("Instrumentation is : ".applicationInstrumentation);

ReservedForFutureUse = "";
poller->instance_instrumentations +=
list(applicationInstrumentation->Name,
snmpAddress,
index,
ReadCommunity,
WriteCommunity,
EngineID,
EngineBoots,
EngineBootTime,
UserName,
AuthProtocol,
ReservedForFutureUse,
AuthPassword,
ReservedForFutureUse,
Context,
AgentVersion) ? LOG,FAIL;

poller->polling_parameters +=
list(applicationInstrumentation->Name,
numeric(PollingInterval),
-1,
numeric(Timeout),
numeric(Retries),
-1) ? LOG,NEXT; //change -1 to TRUE for SNMP trace logging

print("Completed adding instance to accesor");
}
stop();
}
DEFAULT {
line:{..eol}
}
do {
print("Bad input: ".line);
}

Tests

First, I need to create the application objects and the relationship between the host and the MyApplication objects. This can also be automated as part of the discovery process, but that's not our focus right now. I've used the following dmctl commands:

create MyApplication::MySQL

put MyApplication::MySQL::HostedBy Host::ifountainGeneva

Then I run the asl script

sm_adapter -s INCHARGE-AM -DappName=MySQL appLoadAccessor.asl

At this point, you should be able to see the HostedBy and InstrumentedBy relationships of the MyApplicaton object.
Last thing to do is to make sure is there is something that subscribes to the event (otherwise it won't be polled). This is done in the dynamic model tutorial with an import file, but I think specifying the events for MyApplication class in the dxa file for the SAM server takes care of this since it means SAM server subscribes to these events.
So if all this works out as they should, you'll end up with this :)

In the next (and most likely the last) post in this series on Smarts dynamic model, I'll talk about how we can handle the two open issues that are left using our RapidConnector product:

• How to instrument the objects in the Smarts repository using RapidConnector easily, rather than struggling with SNMP instrumentation in dynamic model. Note that this is also an option when SNMP agent is not available.
• How to have a single root-cause notification, rather than two root-cause notifications when the host an application is running on is down. I will also use RapidConnector to make application down notifications "impact' of host down notifications.

So if you're interested in the subjects above, stay tuned. If not, I hope that this series of articles have provided some insight to the world of dynamic model!