Optimizing Servers for Application Performance

Monitoring Server Performance and Activity

One of the basic steps to optimizing server performance is to monitor server performance. To optimize server performance, you can perform a number of tasks, such as:

• Reducing the load of network traffic on the particular server by implementing load balancing strategies.

• Reducing CPU usage

• Improving disk I/O

In order to optimize server performance for whatsoever reason, you need to start by monitoring the server. In most cases, before server monitoring commences, it is common practice to establish baseline performance metrics for the specific server. Baseline performance metrics are established by measuring the performance of a particular server under various conditions, at different times of the day, week, and month, and when the hardware and software configuration changes. Based on the baseline metrics which you define for the server, you would need to optimize the server when performance of the server by far exceeds your baseline metrics.

It is recommended to formulate a thorough server monitoring plan once you have established your baseline performance metrics for a specific server.
A few elements that should be incorporated into a server monitoring plan are:

• Identify those server events which should be monitored.

• Determine and set up monitors and alerts to inspect the events

• Determine whether filters are necessary to reduce the quantity of monitoring information which you want to collect and then configure any necessary filters.

• Establish a logging strategy to log event data that should be analyzed. Alternatively, you can monitor and analyze server activity when it occurs.

• View and analyze data collected by events in the Performance console.

Another key component to monitoring server performance is to identify bottlenecks. A bottleneck can be described as a condition which is created by a particular resource which in turn prevents other resources from operating optimally.

Bottlenecks occur when:

• Certain settings are not configured correctly.

• A resource is faulty and not functioning as it should be.

• Insufficient resources exist, and a resource is being overused.

• Where many instances of the same particular resource exist, the resources may not be handling load equally or efficiently.

• A particular application(s) is hogging a resource.

Through monitoring and analyzing the event data that is collected, you can identify which resources are slowing the system down and causing poor system performance. Server and system performance is usually impacted by the following:

• Resources are configured incorrectly which are causing resources to be intensely utilized.

• Resources are unable to handle the load it is configured to handle. In this case, it is usually necessary to upgrade the particular resource or add any additional components that would improve the capability of the resource.

• Resources that are malfunctioning impair performance.

• The workload is not configured to be evenly handled by multiple instances of the identical resource.

• Resources are ineffectually allocated to an application(s).

The subsystems which should be monitored when monitoring system performance are listed below. These subsystems should be monitored and optimized to tune the server for application performance are:

• Memory subsystem

• Network subsystem

• Processor subsystem

• Processes subsystem

• Disk subsystem

Monitoring system or server performance in Windows Server 2003 remains fairly unchanged when compared to the tools and features utilized in Windows 2000.

The main enhancements introduced in Windows Server 2003 specific to performance monitoring are listed below:

• System Monitor: With Windows Server 2003 System Monitor has been improved so that you can view multiple log files concurrently.

• Command-line utilities: A few new command-line utilities have been added:

  • Logman: The utility can be used to schedule and administer performance monitoring sessions.

  • Relog: Reglog can be used to change the data collected as a counter log to either of the following formats:

    • Binary format

    • SQL format

    • Text-TSV format

    • Text-CSV format

  • Tracerpt: The utility can be used to create a trace analysis report for data which was collected via an event trace provider.

  • Typeperf: For writing counter data in the current command window to a counter data log file.

• Performance Logs And Alerts: The tool has been improved in a number of ways:

  • A few new file formats are supported in Windows Server 2003.

  • You can store log files more than 1GB.

  • You can add collected performance data to an existing file.

  • By using an Open Database Connectivity (ODBC) connection, you can log data to a SQL database.

Utilizing System Monitor
You can use System Monitor to collect and analyze performance data for both the local computer and remote computer, and to track various processes running on a Windows Server 2003 or Windows 2000 system. The System Monitor utility is located within the Performance MMC snap-in.
You have be a member of one of the security groups listed below to utilize System Monitor to monitor server performance:

• Administrators

• Performance Log Users

• Performance Monitor Users

• Server Operators

A few key activities that can be performed using System Monitor and the main features of System Monitor are listed below:

• View and monitor real-time performance data or view data from a log file.

• Choose which data you want to collect by selecting between various objects and counters.

• Collect data from multiple computers simultaneously.

• Choose the format or view in which you want to examine collected data:

  • Graph view

  • Report view

  • Histogram view

• Monitor processes and components that you want to optimize

• Monitor the results of your optimization efforts

• View trends in workloads and the effect being realized on resource usage.

• Plan for upgrades

How to select the counters you want to monitor

1. Under the Administrative Tools menu, select Performance to open the Performance console.

2. In the left pane, select System Monitor.

3. To view current activity, click the View Current Activity button on the Performance Monitor toolbar.

4. To add the counters which you want to monitor, click the Add button on the Performance Monitor toolbar.

5. The Add Counters dialog box opens, displaying a number of fields.

6. If you want to monitor the local computer, click the Use Local Computer Counters option.

7. If you want to monitor a different computer, click Select Counters From Computer, and choose the server which you want to monitor.

8. Select the type of performance object you want to work with from the Performance Object drop down list.

9. If you want to monitor each counter associated with the particular performance object, select the All Counters option.

10. If you want to choose specific counters, choose the Select Counters From List option and proceed to select the counters.

11. If you want to monitor all counter instances, click the All Instances option.

12. If you want to select specific counter instances, click the Select Instances From List option and then choose the counter instances to monitor.

13. Click Add to add the counters for the performance object.

14. Click Close after you have added all the counters which should be tracked.

Utilzing Performance Logs And Alerts
By using Performance Logs And Alerts included in the Performance console, you can track the performance of a server by creating counter logs, trace logs, and defining alerts.

• Counter Logs: Counter logs collect data on the selected counters once a predefined interval has passed. A few features of counter logs are listed below:

  • Collect performance data on the activities of system services.

  • Collect performance data on hardware usage.

  • Manually log data.

  • Schedule data logging to both commence and stop automatically

  • You can view collected data via System Monitor.

  • Export collected performance data to a database or spreadsheet.

• Trace Logs: Trace logs differ to counter logs in that trace logs monitors data constantly and then records performance data when a specific event occurs. The collected data is written to a trace log file.

• Alerts: You can configure alerts to be generated when a predefined counter reaches, exceeds or falls beneath a defined threshold or value. When you configure alerts, you can specify that the following actions occur when an alert condition is met:

  • A message is sent to a user through the Windows Messenger service.

  • A log file is created.

  • An application or program is executed.

How to start Performance Logs And Alerts

1. Click Start, Administrative Tools, and then click Performance.

2. Expand the Performance Logs And Alerts node.

3. You can now manage existing counter logs, trace logs and alerts; and configure new counter logs, trace logs and alerts, or delete existing counter logs, trace logs and alerts.

How to manage counter logs and trace logs

1. Open the Performance console

2. Expand the Performance Logs And Alerts node.

3. To configure a counter log, select Counter Logs OR select Trace Logs to configure and manage trace logs.

4. The right pane would display or list any existing logs.

   • A red log symbol signifies that logging has stopped

   • A green log symbol signifies that logging is active.

5. You can manage existing counter and trace logs by selecting the appropriate log entry in the right pane, and then selecting one of the following options from the shortcut menu:

   • Properties, to open and view the properties dialog box of the particular counter or trace log.

   • Delete, to delete the counter or trace log

   • Start, to start logging

   • Stop, to stop current logging

   • Save Settings As, to save the log configuration as a Web page

6. You can create a new counter log, or trace log by selecting the log which you want to create, clicking in the right pane and then selecting New Log Settings from the shortcut menu.

How to create a new counter log

1. Open the Performance console

2. Expand the Performance Logs And Alerts node.

3. Right-click Counter Logs, and then select New Log Settings from the shortcut menu.

4. When the New Log Settings dialog box opens, enter a name for the new counter log, and then click OK.

5. The counter log file Properties dialog box opens next, with the General tab displaying. This is the location where you define the objects and counters which you want to track.

6. If you want to add counters for a specific performance object, click the Add Objects button. When the Object dialog box opens, select the objects which you want to add. In this case, all counters of the selected object will be tracked or logged.

7. If you want to monitor only specific counters, click the Add Counters button. When the Select Counters dialog box opens, select the specific counters that should be added.

8. In the Sample Data Every field, set the sample interval when data should be collected, and choose a time unit.

9. In the Run As box enter the Name of the account under which the log should run. Click Set Password to enter the password associated with the account in the Set Password dialog box.

10. Click the Log Files tab to specify the type of log file which should be created. Counter logs are by default saved as binary files in the %SystemDrive%PerfLogs directory.

11. In the Log File Type drop down list box, select between the following options:

    • Text File (Comma Delimited)

    • Text File (Tab Delimited)

    • Binary File

    • Binary Circular File

    • SQL Database

12. Select the End File Names With checkbox if you want to include a suffix for each new file which is created when the counter log runs.

13. In the Start Numbering At box, enter the first serial number for the log if an automatic numeric suffix is utilized.

14. You can specify a comment for the counter log in the Comment box.

15. Click the Schedule tab to indicate when logging should start and stop. You can configure logging to either start manually or automatically at a specific date, the action that should occur, and when the log file should be closed.

16. After you have configured the logging schedule, click OK.

How to create a new trace log

1. Open the Performance console

2. Expand the Performance Logs And Alerts node.

3. Right-click Trace Logs, and then select New Log Settings from the shortcut menu.

4. When the New Log Settings dialog box opens, enter a name for the new trace log, and then click OK.

5. The trace log file Properties dialog box opens next, with the General tab displaying. This is the location where you define trace log properties.

6. To trace operating system events, click the Events Logged By System Provider option and then select the operating system events that should be traced.

7. To trace nonsystem providers, click the Nonsystem Providers option, and then click Add. When the Add Nonsystem Providers dialog box opens, select the provider you want to track.

8. In the Run As box enter the name of the account under which the log should run. Click Set Password to enter the password associated with the account in the Set Password dialog box.

9. Click the Log Files tab to specify the type of log file which should be created.

10. In the Log File Type drop down list box, select between the following options:

    • Sequential Trace File

    • Circular Trace File

11. Click the Configure button to set the location, filename, and log file size.

12. Click the Schedule tab to indicate when logging should start and stop. You can configure logging to either start manually or automatically at a specific date, the action that should occur, and when the log file should be closed.

13. Click the Advanced tab to configure the buffer settings for the log file.

14. Click OK.

How to create an alert

1. Open the Performance console

2. Expand the Performance Logs And Alerts node.

3. Right-click Alerts, and then select New Alert Settings from the shortcut menu.

4. When the New Alert Settings dialog box opens, enter a name for the new alert, and then click OK.

5. The alert file Properties dialog box is displayed next. This is the location where you can configure alert properties.

6. Click the Add button on the General tab to specify which counters you want to track.

7. Specify the condition when the alert should be generated for the counter.

8. In the Run As box, enter the name of the account which will be utilized to generate the alert.

9. Click the Action tab to configure the action that should occur when the alert is generated:

   • Log An Entry In The Application Event Log; an entry will be logged which you can view through Event Viewer.

   • Send A Network Message To; results in a message being sent.

   • Start Performance Data Log; causes an existing counter log to be run.

   • Run This Program; causes a command file to run.

10. Click OK.

Utilizing Task Manager
Task Manager is the main tool utilized to manage applications. To open Task Manager, use one of the following methods:

• Right-click the taskbar and then click Task Manager on the shortcut menu

• Enter taskmgr at the command prompt.

• Press Ctrl+Alt+Del and select Task Manager.

The Application tab of the Task Manager tool displays the status of applications which are currently running. The management functions which you can perform by clicking the buttons located at the bottom of the Application tab are:

• To stop a particular application that is currently running, select the application and click the End Task button.

• To switch to an application to activate it, select the application and click the Switch To button.

• To start a new application, click the New Task button and enter the command to execute the application.

You can perform various application specific management functions by right-clicking an application and then selecting between the available options on the shortcut menu:

• Switch to an application to activate it.

• Minimize an application and maximize an application

• Tile and cascade the application

• End an application

• Move the application to the front of the display.

• Navigate to the associated process on the Process tab.

Monitoring Memory Usage

When you encounter system performance issues, the main component that should be monitored is memory usage. Through System Monitor, you can configure a number of counters of the memory object which you want to monitor.

The most important performance counters which you should monitor to detect memory issues are:

• MemoryAvailable Bytes: The counter indicates what your available memory capacity is. To efficiently run a server, you need to have at least 4MB of memory available. When the memory drops below 4MB, you might need to consider immediately adding more memory.

• MemoryPages/Sec: With a recommended counter threshold of 20, this counter indicates the rate at which pages are written to disk, or read from disk. The counter should generally be beneath 20.

The performance counters which should be monitored to detect bottlenecks or memory leaks are:

• MemoryPages/Sec: This counter was discussed previously.

• MemoryCommitted Bytes: This counter indicates the number of committed bytes of virtual memory on the system, and should be monitored together with the MemoryAvailable Bytes counter if you suspect that a memory leak exists.

• MemoryPool Nonpaged Bytes: This counter indicates the number of bytes allocated to the nonpaged pool for those objects which cannot be written to disk. You might need to supplement your existing memory if this counter value is high.

• MemoryPool Nonpaged Allocs: The counter indicates the number of calls to allocated space within the nonpaged pool.

• ServerBytes Total/Sec: This counter indicates how busy the server is, and displays the number of bytes which the server has sent to and received from the network. A drastic increase in this counter value could indicate that you need to add more memory.

• ServerPool Paged Bytes: This counter tracks the number of bytes of pageable memory being utilized.

• ServerPool Nonpaged Bytes: The counter tracks the number of bytes of nonpageable memory being utilized.

One of the most common issues that affect server performance is memory leaks which are caused by incorrect application code. A few strategies which you can use to rectify this are:

• Ensure that your memory intensive applications are executed on those higher performing computers.

• Use Task Manager to determine the minimum memory requirement necessary for the applications to execute, and then ensure that the available memory surpasses this figure.

• To enable faster disk access between the disks, configure multiple paging files on multiple disks.

• Ensure that the paging file size is 1.5 times the physical RAM.

How to optimize or tune memory when you suspect a memory bottleneck

• Verify that the paging file size correct

• Increase the physical memory installed on the computer.

• Steer clear of having the paging file on the same partition containing system files.

• Consider running less memory-intensive applications

• Consider creating multiple paging files on multiple disks

The counters which should be monitored to detect low memory situations are:

• MemoryAvailable Bytes: The counter indicates what your available memory capacity is. To efficiently run a server, you need to have at least 4MB of memory available.

• MemoryCache Bytes: Tracks the number of bytes which the file system cache is utilizing.

• Physical Disk% Disk Time; and Physical Disk Avg. Disk Queue Length: A memory shortage would exist if an increase in queue length is not followed a decrease in the Memory Page Read/Sec counter value.

The counters which should be monitored to detect excessive paging are:

• Paging File % Usage: Because paging files are used to store pages of memory, and are shared by processes, the paging file can cause a bottleneck. The threshold value of this counter is 99 percent.

• Paging File % Usage Peak: You should consider increasing the size of the page file if the value of this counter moves towards the maximum paging file setting.

• Physical Disk Avg. Disk Sec/Transfer: The counter indicates the average disk transfer in seconds.

• Memory Pages/Sec: With a recommended counter threshold of 20, this counter indicates the rate at which pages are written to disk, or read from disk. The counter should generally be beneath 20.

How to optimize the paging file for better performance

• Increase the size of the paging file if you have adequate disk space

• Distribute the paging file over multiple hard disks.

Monitoring Network Activity

When you monitor network objects, you would basically be tracking network traffic and examining the usage of server resources. Flaws in the network architecture can lead to problems with server memory.
The important counters which should be tracked to monitor the performance of the network for possible bottlenecks are:

• Network InterfaceBytes Total/sec; Bytes Sent/sec; Bytes Received/sec: These counters indicate the manner in which your network adapters are performing in relation to network traffic.

• ServerBytes Total/sec; Bytes Received/sec; Bytes Sent/sec: This set of counters indicates the manner in which the server is utilizing the network to send data and receive data.

• ServerPool Paged Peak: This counter shows the amount of physical memory and the maximum paging file size. A threshold of the amount of physical RAM is fine.

How to solve network performance issues are:

• Unbind unnecessary and infrequently used network adapters, and then upgrade your network adapters to higher performing adapters. This would greatly increase performance.

• You can increase file sharing throughput by deploying multiple network adapters.

• Consider placing domain users on the same subnet. This could assist in reducing and preventing unwanted replication traffic.

• If you are using multiple protocols for network connections, place the frequently utilized protocol at the top of the protocol list. This would assist in reducing the average connection time.

Monitoring Processor Activity

If you want to determine how processors are being utilized on the server, you should monitor processor activity. The factors that need to be kept in mind when monitoring processor activity are:

• The role of the server

• The type of work being performed on the server

Processor bottlenecks are typically caused by:

• Excess demand put on the processor by CPU intensive applications

• Excess interrupts caused by device drivers, and network and disk components

The important performance counters that should be tracked to monitor the processor element of the server for bottlenecks are:

• Processor% Interrupt Time: The counter indicates how often devices such as network adapters and disk drives generate interrupts by showing the time duration for which the processor receives and services hardware interrupts in the sample interval.

• Processor Interrupts/Sec: This counter shows the number of interrupts which are being received from devices. An increase in the counter's value should be accompanied by an increase in system activity. If not, you could have an existing hardware issue that needs to be resolved.

• System Processor Queue Length: This counter shows the number of requests waiting in queue to be processed. If you constantly have a queue length of over 10 for a processor, you may have an existing bottleneck.

• Server Work QueuesQueue Length: The counter shows what the queue length of the Server Work queue is. An acceptable value for this counter is four. If there are constantly over four items in the queue, it could be indicative of processor congestion.

The counters which you should monitor to detect usage issues are:

• Processor% Processor Time: This counter should be tracked if you want to detect a process which is utilizing over 85 percent of the processor time. In this case, you might have to consider either upgrading the processor, or installing an additional processor.

• Processor% User Time: This counter indicates the non-idle processor time percentage which takes place in user mode. If the value of the counter is constantly high, consider upgrading the processor, or installing an additional processor.

• Processor% Privileged Time: This counter indicates the non-idle processor time percentage assigned for operating system components and hardware manipulating drivers. A high value of this counter could be indicative of a hardware device failing.

How to optimize the processor subsystem

• In cases where multithreaded programs are being run, you should:

  • Replace any failing processors

  • Upgrade your existing processors to faster processors.

  • Add any necessary additional processors.

• You should spread programs effectively over your existing servers so that the workload is spread efficiently.

• Consider scheduling programs to run during off peak hours.

Monitoring Disk Activity

Monitoring disk usage and activity is important if you want to optimize the performance of the server, or more efficiently balance the load of your servers. It is recommended to log performance data to a different computer or disk so that it does not affect the data which you are monitoring.

The important counters which you should monitor to track disk performance of a server are:

• Physical Disk Current Disk Queue Length: The counter indicates the number of requests which are in queue for disk access. The counters should be consistently at 1.5 to 2 times the number of spindles which comprise of the physical disk.

• Physical Disk % Disk Time and % Idle Time: These counters indicate the percentage of time that a drive is active, and the percentage of time for which the drive is idle. The recommended threshold for both counters is 90 percent.

• Physical Disk Avg. Disk Sec/Transfer: The counter indicates the time duration which the disk uses to deal with a request. A counter value of 0.3 seconds or above means that a high average disktransfer time exists.

• Physical Disk Disk Reads/Sec and Disk Writes/Sec: These counters show the speed at which data is written to disk, and read from disk. The information of these counters could assist you in more efficiently balancing the load of your servers. A lengthy delay could be indicative of a hard disk issue. When usage is close to the defined transfer rate of the physical disk, you might need to consider decreasing the load of the server. You could also upgrade to RAID.

• Physical Disk Bytes/sec: The counter indicates throughput of disk activity

• Physical Disk Avg. Disk Bytes/Transfer: This counter tracks the average number of bytes which are either moved to or from a disk when disk reads and writes occur. A program is typically accessing the drive ineffectually when the counter's value is over 2KB.

• LogicalDisk % Free Space: The counter shows how much available free space exists on the disk in relation to total disk capacity.

How to solve disk performance issues and optimize disk activity on a server

• If necessary, install additional disks.

• If you upgrade a disk to a faster one, upgrade both the disk controller and bus simultaneously.

• To balance the workload, distribute applications between your servers.

• You can also distribute your applications over multiple disks.

• To improve disk access, install the most up-to-date driver software for the host adapters.

• To improve throughput, create striped volumes on a number of physical disks.

• To optimize disk performance and minimize disk access time, run Disk Defragmenter frequently.