Submitter Mode in BMC ITSM

Today, we can see about 'Submitter' Mode in BMC AR System

Submitter Mode:

Two Options:

 Locked: The value of ‘Submitter’ can never change

Changeable: ‘Submitter’ can change

Its Use:

• Ability for the submitter to edit ticket/data they submitted, both in and out of the BMC Remedy applications
• Development workflow that has been designed to accommodate your choice

Please note that this is a brief note and you can go through the bmc docs for in-depth info about this topic.

Discuss about 112 field - Assignee Group

Today I am going to brief about '112' field in BMC AR System

Field 112: The “Assignee Group” Field

                                                         Used to enforce row level permissions

                                                          BMC Remedy permissions are cumulative

                             If “Public”is ever in field 112 everyone will be able to see it

To enforce row level permissions all that is needed is:

                                                               i.      –Add field 112 to a form

                                                             ii.      –Add workflow to push the Group ID of any group that should have access

Please note that this is a brief note and you can go through the bmc docs for in-depth info about this topic.

ServerGroup : Message not in catalog -- message number = -1 (ARERR -1)

ServerGroup : Message not in catalog -- message number = -1 (ARERR -1)

If you are seeing the above error, it may be because the DB server may gone down and up after some time. Check with DB team and also look out for arerror logs. 

Though the DB is back, arserver maynot have re-established the connection with DB properly. Restart of AR Server would help in establishing the connection back.

Read user not able to answer a Survey in SRM:Survey form.

Read user not able to answer a Survey in SRM:Survey form.

When an Incident is closed a Survey is sent to the Requester of the Incident.

If this requester has a read license, then he/she cannot answer the Survey as it gives "You do not have write license" on some fields like z1D_Question1_Rating, z1D_Question2_Rating, User_Action, etc.

Error happens when they submit the survey. We  get logged in and answer survey questions fine. Error happens after completing survey questions, on submit.

You do not have write license (AREE 8932)

You do not have write access to field:

z1D_Question9_Rating (ARERR 330)

What I did to resolve this issue

Remove the "SRM:Survey" form from active link "SRM:SHR:Save_CreateEntryOnForm_Submit" and save.

Flush the cache on your midteir server.

Note: Flushing cache will slow everything up on midtier as all forms and workflow will then have to recache

Notification for surveys has an incorrect url in the message body

Notification for surveys has an incorrect url in the message body

IN ITSM, when a survey is sent to the end user the survey contains the url as http://<midtierserver>/arsys/servlet/ViewFormServlet?form=SRM:Survey&view=RequesterAnswer&server=<arserver>

Verify whether the correct url is configured. if not modify it as follows.

Open the form SYS:Attachments in the user tool and search for 'Description' = "URL for Survey".  On this record modify the field with the url.  Change <midtierserver> to your value and <arserver> to your value.  Save the record.

AR Web Service Time out

AR Web Service is timing out

BMC Remedy AR System Server 7.6.04

Encountering the following error: Message not in catalog; Message number = 9130; 

nested exception is: java.net.SocketTimeoutException: Read timed out (ARERR 9130).

The above error is generated when filter consumes the wsdl 


Solution I found for the above issue

Checked pluginsvr_config.xml file. It should be under the location

<BMC Software Installation DIR\<server_name>\pluginsvr

There is a configurable time out parameter.

<plugins>

<plugin>

<name>ARSYS.ARF.WEBSERVICE</name>

<pathelement type="location"><BMC Software Installation DIR/<server_name>/pluginsvr/websvcjava7604_build002.jar</pathelement>

<classname>com.bmc.arsys.ws.plugin.WSPlugin</classname>

<userDefined>

<policyConfigDir><BMC Software Installation DIR/<server_name>/pluginsvr</policyConfigDir>

<!-- This configuration allows you to set the time-out for

webservices plugin. The value of timeout depends on following rules:-

* the timeout value finally considered would be minimum of the

filter-api-timeout and timeout set here.

* timeout value for plugin would always be less than

filter-api-timeout value (as configured in ar.cfg).

* filter-api-timeout value can max be of 300 secs.

-->

<timeout>40</timeout>

</userDefined>

</plugin>

<plugin>

 By default it is 40 seconds. Increase this value to 300 seconds & restart Java Plugin Server.

Also make sure Filter-API-Timeout & Server-Default-Plugin-Timeout also set to 300 seconds. Maximum valid value is 600 seconds.

Web Service Registry Location cannot be empty (ARERR 9133)

If you encountered the below errors in arerror.log

Web Service Registry Location cannot be empty (ARERR 9133)

To resolve ARERR 9133  try the suggested workaround below:

Go to the "AR Server Information" >> WSC Registry Integration.

Please fill all the information, Save, and click on Update Registry

Registry Location - http://<server-name>:port/uddi

Registry Admin User:  admin; Registry Admin Password: admin ( default password )

Restart the AR Server services and check issue is resolved.

Timeout during database query -- consider using more specific search criteria to narrow the results, and retry the operation ARERR - 94

Timeout during database query -- consider using more specific search criteria to narrow the results, and retry the operation ARERR - 94

or

Timeout during data retrieval due to busy server -- retry the operation ARERR - 93

This error is caused by database server overhead due to unqualified search and it should be disabled. To disable unqualified search perform the following:

a) Go to Server Information form.

b) Go to Configuration tab

c) Disable "Allow Unqualified Searches"

If the issue still occurs, Enable the server logs and look out for the code that took time to retrieve data from DB. Fine Tune the code, if necessary.

Plugins in AR Server & Fine Tuning of Plugin Parameters

We have been looking into some of the issues in our blog so for. Today we are going to look into the plugins in AR Server and how to fine tune.

There are two plugin server technologies, C and Java.

Some plugins will run as C plugins under arplugin (arplugin.exe ) executable.

Some plugins will run as Java plugins. The Java Plugin server runs as a Java Virtual Machine.

In a default ITSM environment there are 4 Java Plugin servers installed; the primary Java Plugin server, the Full-Text Search Engine Plugin server, and 2 CMDB Java Plugin servers.

 The C Plugin server is configured through the ar.cfg file, while the Java Plugin servers are configured through pluginsvr_config.xml file.   The primary Java Plugin server is located under the <ARSystem_Install_Dir>\pluginsvr and the pluginsvr_config.xml

there are 3 types of plugins, AREA, ARDBC, and ARF in ITSM.

1) AREA is used for Authentication. Example: LDAP, SSO, or customer plugins.

2) ARDBC is used for Vendor forms such as the Overview Console

3) ARF is used to call Filter APIs such as with the CAI plugin and the Web Services plugin.

To Configure AR Server for the Plugins, one has to look out for the following details in ar.cfg

Plugin-ARDBC-Threads:

Plugin-Filter-API-Threads:

Plugin-AREA-Threads:

Plugin-Loopback-RPC-Socket:

Private-RPC-Socket:

Server-Plugin-Default-Timeout:

And in pluginsvr_config.xml

<numCoreThreads>5</numCoreThreads>

<numSelectorThreads>2</numSelectorThreads>

Note: Before making any changes to the above properties, make sure the following details are taken care

1) Whether the environment has enough bandwidth for plugins, hardware to accommodate the changes.

2) Backup of ar.cfg file

3) Backup the pluginsvr_config.xml

Following numbers are reasonably good in an environment

For ar.cfg

Plugin-ARDBC-Threads: 4 12

Plugin-Filter-API-Threads: 4 12

Plugin-AREA-Threads: 4 12

Plugin-Loopback-RPC-Socket: 390626

Private-RPC-Socket: 390626 4 12

Server-Plugin-Default-Timeout: 180

For pluginsvr_config.xml

                <numCoreThreads>30</numCoreThreads>

                <numSelectorThreads>2</numSelectorThreads>

• The number of selector threads should always be 2 regardless of what it’s running.

• For FTS, The number of core threads should be the same as the high watermark for list threads on the AR server. Look at the ar.conf/ar.cfg file (“Private-RPC-Socket: 390635   8 12” ).         If the value is 12 then set the <numCoreThreads>12</numCoreThreads>

• For the Main Java Plugin Server, since it uses Fast and List Operations, this should be the high watermark of Fast/List/Private threads. Setting the Core threads to 30 is recommended for the main Java Plugin server

• For the 2 CMDB Java Plugin servers (NE and Shared) leave the CoreThreads set to the default of 5

Data type mismatch error or Operation cancelled due to error

If you are seeing
1) Data type mismatch error in arjavaplugin.log "(Data type mismatch. Vendor field: 1000000218 is of data type)"
2) Dispatch : Operation cancelled due to error (ARAPPERR 4502) in arerror.log

Ignore this error, as this is known issue and has no impact on the application. This issue got fixed in the later releases.(7.6.04 SP2 or later)

AR Server License is invalid or Reached the maximum no. of DB entries

Although the license key is valid, we are getting the below error message“You have reached the maximum no. of DB entries. Plz contact BMC Sales team to purchase license”

After thorough analysis at the BMC end, working the Network team helps in identifying the issue.  

AR Server have 2 NIC Adapters with NIC 1, NIC2. While requesting for the AR license with BMC, we got the physical address of NIC 1 (using ipconfig /all) and AR License is applied.

Recently, one of the Network adapters (NIC 1) got disabled, which is the reason for the above error.

Network Team enabled the NIC 1 adapter, which resolves the above issue. 

But after enabling the NIC 1 Adapter, restarting the BMC Server throws plug-in server error as below and is not able to connect to plugin server.

“(9998) ARSYS.ARF.FTS : RPC: Miscellaneous tli error - System error (Connection refused)”

Further investigating the issue, we have identified that the plugin server is automatically pointing to the ip address of the NIC2 Adapter(when the NIC 1 was disabled)

I requested the Network team to disable the NIC 2 adapter temporarily (ie. Remapping the Ip address of the plugin server automatically) and restarted the BMC Services. 

Neither the plugin error nor the license error has appeared after that.

OverView console plugin not responding problem; ARERR [8760] Cannot establish a network connection to the AR System Plug-In server : localhost (9999)

If you face the below issue, while login to Mid-tier.

Encounter the following error when logging into Mid-tier:

"Cannot establish a network connection to the AR System Plug-In server : localhost (9999) : RPC: Miscellaneous tli error - System error (Connection refused) (ARERR 8760)"

ARS 7.6.04, Windows 2008 64 bit, LP: BMC Remedy AR System Server 7.6.04, DR: AR System Server 7.6.04, DM: FTS 7.6.04

at com.remedy.arsys.backchannel.TableEntryListBase.doQuery(Unknown Source)

at com.remedy.arsys.backchannel.TableEntryListBase.getAndEmitTable(Unknown Source)

at com.remedy.arsys.backchannel.GetTableEntryListAgent.process(Unknown Source)

at com.remedy.arsys.backchannel.NDXRequest.processRequest(Unknown Source)

at com.remedy.arsys.backchannel.NDXRequest.<init>(Unknown Source)

at com.remedy.arsys.backchannel.EntryListBase.<init>(Unknown Source)

at com.remedy.arsys.backchannel.TableEntryListBase.<init>(Unknown Source)

at com.remedy.arsys.backchannel.NDXGetTableEntryList.<init>(Unknown Source)

at com.remedy.arsys.backchannel.GetTableEntryListAgent.<init>(Unknown Source)

at com.remedy.arsys.backchannel.NDXFactory.handleRequest(Unknown Source)

Perform the below steps 

1. Confirm if IPv6 is enabled in the Windows 2008 server box by running "ping HOSTNAME" command in the command prompt window.
2. If you have IPv6 enabled on a host, you may encounter problems in AR connecting to Java plugin server or Java plugins connecting to AR server.
3. Run "netstat -nao|findstr 9999" command, which will help to see plugin port number "9999" is listening or not.
4. If above all conditions are met add the following entry to the java line in armonitor.cfg file.

          Note: Before make the changes in armonitor.cfg file, make sure you take the backup of it.

               -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv6Addresses=false

     5. Restart AR Server again. 

• -Djava.net.preferIPv4Stack=true
  -Djava.net.preferIPv6Addresses=false
  For example, if the original entry is:
  “C:\Program Files (x86)\Java\jre6\bin\java.exe” -Xmx512m -classpath “D:\Program Files\BMC Software\ARSystem\pluginsvr; D:\Program Files\BMC Software\ARSystem\pluginsvr\arpluginsvr7604_build002.jar”
  The new entry should be:
  “C:\Program Files (x86)\Java\jre6\bin\java.exe" -Xmx512m -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv6Addresses=false -classpath “D:\Program Files\BMC Software\ARSystem\pluginsvr; D:\Program Files\BMC Software\ARSystem\pluginsvr\arpluginsvr7604_build002.jar”

"File not found" or "Caught Exception" while loading Request Entry Console on Mid-tier

If you encounter any of the below issues listed, while opening the Request Entry Console or any other form in Mid-Tier

• Request Entry Console is not appearing. Getting Error: 500 Internal Server. File not found.
• On Page Load, getting caught exception $ARKeywords$ is null or not an object.
• ‘Request on behalf of’ on Request Entry Console is throwing ‘caught exception’ error or searching all the records.

To resolve these issues, Do the following steps

1) Clear the browser cache (Browsing history / Temporary Internet Files)

2) Clear out temp files under TEMP directory. TEMP here usually refers to "Documents and

Settings\Users<windowsloginuser>\Local Settings\Temp".

3) Flush the Mid-tier Cache

4) Login again.

How to Capture Client Side ARAPI Logs

Follow the steps to capture the Client Side ARAPI Logs

• Stop the JSP engine
• Please copy the file named <Midtier>\WEB-INF\lib\arsys_sample.xml to <Midtier>WEB-INF\classes\arsys_api.xml
• Modify the line

                 <apiRecording>0</apiRecording>
           to
                 <apiRecording>31</apiRecording>

• Restart your JSP engine
• Wait for prefetch operation to finish if enabled
• Access and reproduce the error
• Look for the below logs

                   Service<ARSERVER>#_arapires.log
                   <User><ARserver#_arapires.log 

These logs will be located on yout JSP engine installation directory. If not, search in entire system. ( maybe under "system32" directory )

Analyze these logs and lookout for any issues

Home Page or Console view Icons are not appearing in Mid-Tier

Home Page or Console view Icons are not appearing or displaying properly in Mid-tier on 7.6.04 SP2

To fix this issue, follow the below steps

1)  Login to Mid-Tier Config Page and flush the cache.
2)  Stop the Tomcat Server
3)  Navigate to the 'cache' folder of Mid-Tier Installation Directory and remove all the files.
4)  Start the Tomcat Server
5)  Clear the Browser Cache and Close the Browser
6)  Now, Open the browser and Reload the console again

Mostly, this will help in resolving the issue of Icons not appearing in Mid-Tier.

If the issue is still not resolved, we may dig deeper in to this issue, which I will discuss later.