Siebel Application Response Measurement (SARM)

When an end user clicks a view tab to navigate to a new view, a chain of processes is invoked on different machines in order to handle the request. For example, the Siebel Web Server Extension must connect to the application object manager (AOM) on the Siebel server. The AOM will issue an SQL statement to the Siebel database and the Siebel Web Engine (SWE) will put the view layout and data together and render the view.

In order to be able to make exact measurements to determine performance bottlenecks quickly, Siebel engineers have included instrumentation points in the core program code of all Siebel software components.

This feature is commonly known as Siebel Application Response Measurement (SARM). The following diagram explains how SARM works:

From the above diagram, we can learn the following about Siebel Application Response Measurement:

  1. Requests from the web client are processed on the Siebel Web Server Extension (SWSE) on the web server machine.

  2. The request is passed on to the application object manager on the Siebel server machine.

  3. The application object manager requests data from the Siebel database.

  4. The database returns the data.

  5. When the Siebel code executes, a buffer in the machine's physical memory is filled with information about each single event.

  6. Once the memory buffer is full or a timeout is reached, the buffer's content is flushed to files on the hard disk. These files have a .sarm suffix.

  7. Administrators can use the sarmquery command line utility to convert the binary content of the .sarm files to readable formats such as CSV (comma separated values) or XML. Alternatively, SARM information can be viewed directly at the command prompt.

Next, we will discuss these major administrative tasks related to Siebel Application Response Measurement:

  • Enabling SARM

  • Managing SARM files

  • Using sarmquery to read SARM files

  • Automating SARM data retrieval

Enabling SARM

The default behaviour of Siebel software is not to produce .sarm files. In order to turn on the production of .sarm files, we have to set parameters (for Siebel Servers and their components) or environment variables (for all other variants of Siebel software such as the Siebel Web Server Extension or Siebel clients).

Tips and Tricks 

Even when SARM is "turned off", the memory buffer is always filled with response measurement data. However, the buffer is overwritten once it is full. In the case of an exceptional abort of the program, the buffer is actually flushed to disk to record an image of the last operations (one of which might have led to the error situation). We can find the traces of these flush operations as files with an .fdr suffix (= flight data recorder—in analogy to the voice and data recorders used in aviation). FDR files can be opened with the sarmquery utility as well in order to retrieve information about the final operations of the process that crashed.

Enabling SARM for Siebel Servers and Components

We can use the graphical user interface (GUI) provided by the views in the Administration - Server Configuration screen or the Siebel Server Manager (srvrmgr) command line utility to set the SARM-related parameters. It is recommended to enable SARM only for a specific set of server components rather than for an entire Siebel server in order to keep the performance impact to a minimum.

The following table describes the SARM-related parameters:

Parameter Alias

Description

Example Value

SARMLevel

If set to a value greater than zero, .sarm files are written. A value of 1 produces less information than a value of 2. Value 3 is reserved for Oracle internal use.

2

SARMBufferSize

The amount of physical memory reserved for the SARM buffer (in bytes) for each process instance.

5000000

SARMPeriod

The interval (in minutes) how often the memory buffer is flushed to disk (even if not full).

1

SARMMaxFiles

The maximum number of .sarm files to be produced by each process instance. Once the limit is reached, the oldest file is deleted.

4

SARMFileSize

The maximum size of an individual .sarm file (in bytes).

15000000

SARMThreshold

It is important to set this parameter to the number of milliseconds that define the threshold for "long running" events. Because of the large number of events happening in very short time, we can dramatically reduce the size of the .sarm files by setting this parameter.

100

SARMLogDirectory

It is possible to direct the .sarm file output to any shared or local directory. If not set, the .sarm files will be located in the local log subfolder of the Siebel software installation directory.

\\appsrvr\sarmdata

SARMUsers

A comma-separated list of user account names to limit the amount of data in the .sarm files. If left empty, .sarm data is collected for all users.

sadmin,siebmgmt,adm

The following procedure describes how we can enable SARM for an application object manager component:

  1. Log in to the Siebel Server Manager (srvrmgr) command line using a command similar to the following at the Siebel server's bin directory:

    srvrmgr /g osappeval4 /e SIEBEL_EVAL /u SADMIN /p TJay357D /s Eval_1

The above example command will connect to the Siebel Gateway Name Server on the host osappeval4. The enterprise name is SIEBEL_EVAL, username is SADMIN, and the server to be set at the prompt is Eval_1.

  1. At the srvrmgr> prompt, enter a command similar to the following:

    change param sarmlevel=2,sarmperiod=1,sarmthreshold=100 for comp sccobjmgr_enu

The above example command enables SARM at level 2 for the English Siebel Call Center object manager. The memory buffer's content will be flushed once a minute; only events that took longer than 100 milliseconds will be written to the .sarm files. These parameter changes are effective immediately, so no component or server restart is required.

Enabling SARM for Other Siebel Software Units

Because of the fact that only Siebel servers and their components can read the parameters from the Siebel Gateway Name Server's configuration store, we must use environment variables to enable SARM for other Siebel software units such as the Siebel Web Server Extension (SWSE) or Siebel clients such as Siebel Tools or the Siebel Mobile Web Client.

The following table relates the SARM environment variables to the parameters discussed above. Basically, the environment variable's name can be derived by adding the prefix "SIEBEL_" to the parameter name.

Environment Variable

Related Parameter

SIEBEL_SARMLevel

SARMLevel

SIEBEL_SARMBufferSize

SARMBufferSize

SIEBEL_SARMPeriod

SARMPeriod

SIEBEL_SARMMaxFiles

SARMMaxFiles

SIEBEL_SARMFileSize

SARMFileSize

The following procedure describes how to set the SARM environment variables on Microsoft Windows-based operating systems:

  1. Right-click the My Computer icon and select Properties.

  2. In the System Properties dialog, click the Advanced tab.

  3. Click the Environment Variables button.

  4. In the System Variables area, click the New button.

  5. Enter the correct values in the Variable Name and Variable Value fields and click OK.

  6. Repeat the previous step for each environment variable you need to set.

  7. Restart the application (for example the Siebel Developer Web Client or the web server).

We can verify that SARM is properly enabled by inspecting the log directory of the installation folder of the Siebel software unit. SARM files should appear in this directory after the time specified in the SARMPeriod parameter. The screenshot below shows the Siebel Server's log directory with .sarm files:

The naming convention for sarm files is as follows:

A_<Server component name>_T<Timestamp>_P<Process Id>_N<counter>.sarm

If we used the SARMLogDirectory parameter to specify a different storage location for .sarm files, we must inspect the respective directories.

Managing SARM Files

It is imperative for Siebel administrators to understand how to create usable sets of .sarm files without cluttering the server's hard drives. In environments with hundreds or even thousands of end users, the amount and size of .sarm files can grow out of bounds quickly.

The following techniques ensure that we harvest a reasonable amount of information without affecting the overall system performance more than necessary.

Always set the SARMThreshold parameter to a value of 10 or higher. This avoids too much data being written to the .sarm files. Because SARM is often used to trace long-running and therefore performance-hampering events, we can ignore the events that execute very fast.

Setting the SARMThreshold for example to a value of 100 reduces the amount of data written to the .sarm files by an average of 90%. This can be easily verified by comparing .sarm file sizes at different settings of this parameter.

Adjust the amount of .sarm files for a single process using the SARMMaxFiles parameter. After a short evaluation period, we should be able to determine the ideal number of files. We should always have the amount of files that is needed to cover a typical monitoring period. For example, if we monitor a work day (9:00 am to 5:00 pm) we should set the SARMMaxFiles parameter to a number which ensures that .sarm files for an individual process do not get overwritten before eight hours.

With regard to reading the .sarm files with the sarmquery utility or using the Siebel Diagnostic Tool, the default folder—the log directory of the Siebel software unit—should be kept as the storage location. However, we can use scheduled scripts to copy the .sarm files to an archive folder to avoid loss of data.

Using Sarmquery to Read SARM Files

The sarmquery command line utility is the primary means of converting the binary content of the .sarm files to readable output. We can choose between various output formats such as text, XML, or csv files or we can consume the output directly at the command prompt.

We can locate the sarmquery executable in the bin directory of the Siebel server's or the Siebel Gateway Name server's installation folder. In addition, the Siebel Management Agent uses sarmquery to support the Siebel Diagnostic Tool, which is discussed later in this chapter.

When we enter sarmquery at a command prompt in the Siebel server's bin directory, the utility reveals the basic usage information. The following screenshot shows the output of the sarmquery command in a Microsoft Windows command shell:

It is recommendable to use the information provided to produce text files that contain the information obtained by the -tips, -help, and -macrosyntax options.

For example, a command similar to the following will create a tips.txt file with useful first step information in the Siebel server's bin directory:

sarmquery -output verbose=tips.txt -tips

The above command uses the -output option followed by the verbose key with a value of tips.txt and the -tips option. This command produces a text file of the standard output of the -tips option. We can repeat this command to create a help.txt and macrosyntax.txt file by modifying the above command accordingly.

The main syntax of the sarmquery utility is as follows:

sarmquery -option key=value -input <directory or file>

The sarmquery command line utility accepts several options, which must be preceded by a dash ("-") and followed by a key=value pair. The -input option must be followed by the name of either a single .sarm file or a directory that contains .sarm files. If we use the directory option, sarmquery will scan the given directory and all its subdirectories recursively for .sarm files.

The following table describes the major options and example key=value pairs for the sarmquery utility:

Option

Example Key=Value Pairs

Description

-input

..\log D:\sia82\siebsrvr\bin\log\\appserver\sarmdata sarmdata0001.sarm

The -input option must be followed by a relative or absolute directory path or the name of a single .sarm file. Directories are always scanned recursively and all .sarm files found are taken into consideration.

-aggregate

area

subarea

user

instance

component

time=60

The -aggregate option is followed by the name of one of the various preconfigured aggregate levels. It defines how SARM data should be grouped in the output. Except for the time aggregate, the -aggregate option does not require values for its keys.

-select

area=DBC

subarea=DBC_EXECUTE

starttime="2010-02-20 09:00:00"

user=CCHENG

The -select option allows constraining the output of the sarmquery utility using preconfigured keys.

-histogram

resptime=500,1000,2000

cputime=100,200,500,1000

The -histogram option groups the output along the specified keys and values. The value is a comma-separated list of millisecond intervals.

-output

sarm=sarmdata.csv

fdr=fdrdata.csv

agg=sarmdata.xml

verbose=tips.txt

The -output option is followed by one of the preconfigured key values, which indicate what content to write to the output file specified as the value.

-config

file=macros.txt

macro=myMacro

The -config option allows to read a sarm macro file and executes one of the macros specified in the file.

Each option or key can be abbreviated as long as the abbreviation is not ambiguous. For example, the following is a valid sarmquery command:

sarmquery -i ..\log -agg sub -sel area=dbc

The above example uses the -input option (abbreviated to -i), the -aggregate (-agg) option followed by the subarea (sub) key and the -select (-sel) option. The abbreviation technique might come handy while using the command line but we should consider using the full text when creating scripts or macro files for the sake of better readability.

For a complete list of possible keys, we can consult the help file (generated using the -help option) or the Performance Tuning Guide in the Siebel bookshelf.

In the following, we will explore common scenarios for sarmquery:

  • Specify the start and end time

  • Application performance by area and subarea

  • Time histograms

  • Identify slow performing objects

  • Create SARM output files

Specifying the Start and End Time

In a typical sarmquery scenario, we most probably want to analyze data from a specific timeframe. In order to do so, we must use the -select option for the starttime and endtime keys in every sarmquery call we make.

We can use a command similar to the following to obtain a list of users who logged on in the specified timeframe:

sarmquery -input ..\log -aggregate user -select starttime="09:00:00" -select endtime="17:00:00"

The above command—issued at the bin directory of the Siebel server—will read all .sarm files in the Siebel server's log directory and create an aggregate plot for all users who had a session between 9:00 AM and 5:00 PM. If no date is given, sarmquery takes the current day into consideration. If we wish to specify a date, we must use the same syntax as in the example below:

-select starttime="2010-05-20 09:00:00"

The example specifies 9:00 AM on the 20th of May 2010 as the start time.

The output of the above command is similar to the following:

 

Histogram Of Average Response Times User Name Avg Response(ms) | Average Inclusive Response Time # Calls --------- -----------------+--------------------------------------------------- -------------- ?Unknown? 42.214 | xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 39 AHANSAL 33.992 | xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 7188 SADMIN 6.377 | . 68880

The previous screenshot shows the typical output of sarmquery at the command line. Aggregates are plotted as horizontal bar graphs with the average response time as the scale.

Application Performance by Area and Subarea

When SARM collects event data, it not only collects the timestamp, the user, and response timings but also the area and subarea of the Siebel application infrastructure where the event occurred.

A full list of SARM data collection areas can be found in the Siebel Performance Tuning Guide in the Siebel bookshelf documentation. The following table lists the most prominent areas:

Area

Description

OBJMGR

Events at the object manager layer. Includes business component and business service execution information.

INFRA

SARM data in this area represents the communication of Siebel server software components between each other.

SWEPAGE

This area groups events in the Siebel Web Engine (SWE), which is responsible for the rendering of user interface objects such as applets and views.

SWE

The events in the SWE area contain information on the applet level such as method invocation.

DBC

The DBC area contains events in the database connector layer. It records performance timings for database operations such as fetch, write, and execute.

EAITRANSP

Events in this area allow performance measurement for Enterprise Application Integration (EAI) touch points with external systems.

THRESHOLD

This area groups all events that fall below the value (in milliseconds) of the SARMThreshold parameter.

WORKFLOW

Events related to the execution of Siebel workflow processes.

SCRIPT

Allows performance measurement of the execution of eScript or SiebelVB code.

SEC

Events related to the security management layer. Includes user authentication.

SARM

The timings for SARM operations.

Each area has one or more subareas. We can obtain a comma-separated value (CSV) file listing all areas and subareas using the following command:

sarmquery -output map=map.csv

The above command produces a text file in CSV format, which describes all available SARM areas and subareas.

Analyzing Areas, Subareas and Instances

We use a sarmquery command similar to the following to get a horizontal bar chart, which helps us identify the area in which most of the process time is spent:

sarmquery -aggregate area -input ..\log

The above command reads all .sarm files from the log directory of the Siebel server and produces a chart similar to the screenshot below:

 

Histogram Of Percent Times Area Percent | Percentage Of Total Self Times Response Time(ms} --------- ---------------+--------------------------------------------------- ----------------- INFRA 25.350 | xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 921900.295 OBJMGR 23.219 | xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 844377.543 SWEPAGE 11.079 | xxxxxxxxxxxxxxxxxxxxx 402914.472 SWE 10.453 | xxxxxxxxxxxxxxxxxxxx 380151.163 DBC 8.749 | xxxxxxxxxxxxxxxxx 318163.928 SRB 8.310 | xxxxxxxxxxxxxxxx 302203.140 THRESHOLD 5.881 | xxxxxxxxxxx 213365.061 EAITRANSP 5.669 | xxxxxxxxxxx 206177.766 CSS 0.574 | x 20360.997 SEC 0.481 | . 17488.999 SARM 0.233 | . 8490.305 FSM 0.001 | . 29.043 SCRIPT 0.001 | . 20.267

The above screenshot shows the output of the -aggregate area option. The chart lists all areas (sorted descending by the percentage of process time spent in the area) for which events have been recorded in the .sarm files. In the above example, 25.35% of the total process time recorded by SARM is spent in events that belong to the INFRA area.

We can "drill down" into each area by using commands similar to the following:

sarmquery -select area=objmgr -aggregate subarea -input ..\log

The above command uses the -select option to filter the output for the OBJMGR area and creates a bar chart that shows the share of process time spent for each subarea.

We can further "drill down" on the data. For example to see a list of business services that use the processing time, we use a command similar to the following:

sarmquery -select subarea=objmgr_bsvc_invoke -aggregate instance -input ..\log

The above command uses the -aggregate instance option. The instance key typically provides the name of the object involved. In the case of the subarea objmgr_bsvc_invoke, business service names are shown in the instance field.

Time Histograms

A common server monitoring scenario is the retrieval of application response data over time. The data contained in the .sarm files can be interpreted in this way using the -aggregate time option. A command similar to the following produces a histogram of the application response time for a specific day:

sarmquery -input ..\log -aggregate time=10 -select starttime="2010-02-18 00:00:00" -select endtime="2010-02-19 00:00:00"

The above command produces a response time plot at 10-minute intervals (specified by the time=10 key=value pair) over the period specified as the timeframe between the starttime and endtime key values.

Identifying Slow Performing Objects

One of the main purposes of SARM is the quick and easy identification of performance bottlenecks. We have learned above that we can aggrgate the response time information by area, subarea, and instance.

By using the correct subarea as a filter criteria in the -select option, we can create response time histograms for objects such as views, scripts, or business services and identify slow performers.

The following example command allows us to identify slow performing views with a build time over five seconds:

sarmquery -input ..\log -agg instance -select subarea=SWEPAGE_VIEW_BUILD -select selftime=5000

The above command produces a response time histogram for all views that required more than five seconds (specified as 5000 milliseconds in the selftime key value) to build (subarea=SWEPAGE_VIEW_BUILD). The following screenshot shows the output of the example command:

 

Histogram Of Percent Times Instance Name Percent | Percentage Of Total Self Times Response Time(ms) ---------------------------------------- ---------------+---------------------------------------------------- ---------------- Enterprise Server/Server View 15.811 | xxxxxxxxxxxxxxxxxx 24627.537 Home Page View (WCC) 24.529 | xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 38206.952 Server Server/Compgroup View 7.883 | . 12278.708 Server Server/Component/Parameter View 10.266 | xxxxx 15990.643 Server Server/Component/State Value View 12.350 | xxxxxxxxxx 19236.399 Server Server/Component/Statistic View 29.162 | xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 45424.415

Views of the server management screens typically take a longer time to build because data is being fetched from the Siebel Gateway Name Server. The high response times for the Home Page View are resulting from the fact that it is the application's start view.

Similar commands can be issued using other areas such as SCRIPT (to identify slow running scripts) or OBJMGR (to identify slow performing business layer objects) or their respective subareas.

Creating SARM Output Files

The sarmquery command line utility can be used to export the content of the .sarm files to various output formats such as plain text, comma-separated values (CSV), or XML. Typically, these files are loaded into third-party software such as Microsoft Excel to further analyze the SARM data sets. We use the -output option to create SARM output files.

One of the most prominent output commands can be seen in the following example:

sarmquery -input ..\log -output sarm=sarmdata.csv#10000

The above command writes the raw SARM data to a series of CSV files. By specifying the suffix #10000, after the file name, we determine that each CSV file will contain only up to 10000 rows of data.

The CSV files can then be consumed for further analysis in spreadsheet applications such as Microsoft Excel or business intelligence tools such as Oracle Business Intelligence Enterprise Edition. The following screenshot shows a SARM .csv output file opened in Microsoft Excel:

The filter feature of Microsoft Excel allows quick creation of filtered sets of data for easier analysis.

In contrary to the raw data output of the sarm key, we can use the agg key of the -output option to write aggregated information to text or XML files. A command similar to the following yields an XML file containing information aggregated at the area level:

sarmquery -input ..\log -aggregate area -output agg=area_agg.xml

The above command uses the agg key followed by an XML file name and produces an output similar to the following screenshot.

<?xml version="1.0" encoding="UTF-8" ?> - <xml> <sarm version="20041117"/> + <Group type="Area" name="INFRA" pctCount="0.054" pctSelfTime="25.350" pctInclResp="35.387"> - <Group type="Area" name="OBJMGR" pctCount="0.316" pctSelfTime="23.219" pctInclResp="25.099"> - <Statistics> + <Self count="4871"> + <Inclusive count="665"> </Statistics> </Group> - <Group type="Area" name="SWEPAGE" pctCount="0.028" pctSelfTime="11.079" pctInclResp="7.157"> - <Statistics> - <Self count="430"> <TimeFrom>2010-01-27 07:18:08</TimeFrom> <TimeTo>2010-02-28 05:33:00</TimeTo> <SearchableTimeFrom>2010-01-27 07:18:ll</SearchableTimeFrom> - <ResponseTime units="ms"> <Average>937.010</Average> <Total>402914.472</Total> <Maximum>24627.637</Maximum> <Minimum>0.005</Minimum> </ResponseTime>

The benefit of the XML format is that SARM data can be made available to third-party analysis software more easily.

Automating SARM Data Retrieval

In order to provide the Siebel CRM project team with fresh and easily readable performance data, we must take the following into consideration:

  • Ensure that .sarm files are collected and archived according to the monitoring policies of our company. We can use shell scripts to copy .sarm files to safe locations so they do not get overwritten by Siebel server processes.

  • Create shell scripts, which execute the sarmquery commands that produce the most usable output for our project. We can use the SARM macro language to produce more flexible script files.

  • Use a scheduling facility such as the Windows task scheduler to invoke the scripts at a regular basis.

  • Provide SARM output files to third-party analysis systems for better visualization of the data.