Prelude
This post is the final part of a series on Optimizing your server. Check out Part 1 (Apache and Passenger Architecture) and Part 2 (Apache and Passenger Directives) of the series if you have not already. In this post, we will look at gathering metrics for your application and tuning the directives accordingly.

Gathering Metrics
A couple of tools that immediately come to mind when thinking of gathering metrics are top and free for Unix systems. However, these tools are not completely accurate in calculating the actual memory usage, especially in a Passenger setup. A more accurate tool is passenger-memory-stats. Running this will give you an output something like the one below

------------- Apache processes ----------------------
PID   PPID  Threads  VMSize    Private  Name
-------------------------------------------------------------
4143  1        1             129.8 MB  0.2 MB   /usr/sbin/apache2 -k start
7672  4143  1             130.3 MB  0.7 MB   /usr/sbin/apache2 -k start
7673  4143  1             130.0 MB  0.4 MB   /usr/sbin/apache2 -k start
7674  4143  1             130.5 MB  0.7 MB   /usr/sbin/apache2 -k start
7685  4143  1             130.3 MB  0.6 MB   /usr/sbin/apache2 -k start
7697  4143  1             130.4 MB  0.7 MB   /usr/sbin/apache2 -k start
7713  4143  1             130.0 MB  0.5 MB   /usr/sbin/apache2 -k start
7715  4143  1             130.3 MB  0.7 MB   /usr/sbin/apache2 -k start
7716  4143  1             130.0 MB  0.4 MB   /usr/sbin/apache2 -k start
7727  4143  1             130.1 MB  0.5 MB   /usr/sbin/apache2 -k start
7729  4143  1             129.9 MB  0.2 MB   /usr/sbin/apache2 -k start
### Processes: 11
### Total private dirty RSS: 5.58 MB

---------- Passenger processes ---------------------
PID    Threads  VMSize    Private      Name
-------------------------------------------------------------
5446   1             104.2 MB  30.6 MB   Passenger FrameworkSpawner: 2.1.0
7701   1             136.7 MB  50.8 MB   Passenger ApplicationSpawner: /home/admin/public_html/rails_blog/releases/20100708065011
7706   1             197.6 MB  47.7 MB  Rails: /home/admin/public_html/rails_blog/releases/20100708065011
23555  13          80.2 MB   0.5 MB    /usr/lib/ruby/gems/1.8/gems/passenger-2.0.6/ext/apache2/ApplicationPoolServerExecutable 0 /usr/lib/ruby/gems/1.8/gems/passenger-2.0.6/bin/passenger-spawn-server  /usr/bin/ruby1.8  /tmp/passenger_status.4143.fifo
23561  2            57.1 MB   11.0 MB   Passenger spawn server
### Processes: 5
### Total private dirty RSS: 140.73 MB

Here again, the VMSize column is not the metric to go by. Instead, the Total private dirty RSS and the Private column values are what you want to trust. Taking that into consideration, in the above output,

- Apache has a total of 11 processes running currently (Apaches current Concurrency)
- Passenger has a total of 5 processes running currently
- Apache is consuming a total of 5.58 MB (of memory)
- Passenger is consuming 140.73 MB
out of which
- the Passenger Framework Spawner is using 30.6 MB
- the Passenger Application Spawner is consuming 50.8 MB
- the Passenger Spawn Server is consuming 11 MB
- a single instance of the Rails app is consuming 47.7 MB

So the current total usage of memory is 140.73 + 5.58 ~= 146MB.

The memory usage for each of the above processes varies based on load on the server.

Tuning the Directives
So now that we know which numbers to trust and how to interpret them, how can we use these numbers to optimize the server setup? Which directives do we need to tune? If you refer my previous article on Apache and Passenger Directives, you will notice that the main directives that leave room for optimization really are, Apache’s MaxClients and Passenger’s MaxPoolSize. So let’s concentrate on these directives.

PassengerMaxPoolSize
When you’re having a single RoR application running on a dedicated server with Apache and Passenger, it should be possible, with all that we’ve learnt, to come up with a generally applicable formula as a guideline for optimizing PassengerMaxPoolSize. And here’s that formula :

(total private dirty RSS for Apache processes) + (total private dirty RSS for Passengers spawn servers) + (PassengerMaxPoolSize * Private dirty RSS per app instance) should be = 80 to 85% of server memory (with a leeway of 15 to 20% for stuff like mysql, SSHing into the server, etc.)

Use this as a guideline, a starting point. At this stage, you need to perform some ab tests to determine how well this setting holds out for you. Running exhaustive ab tests can get quite complicated in itself. The Load Testing Episodes of the Scaling Rails Series are a great resource for that. While you perform your ab tests, be sure to run passenger-memory-stats on your server to observe Apache’s concurrency while it serves your tests. The crux of this exercise, is to determine the average response time of your application, and Apache’s concurrency.

Apache’s Concurrency is the number of Apache processes that are alive at any given point in time. This is defined by the Apache directives – StartServers, MinSpareServers and MaxSpareServers.

MaxClients
Consider this,

“… when an HTTP request comes in to Apache, Passenger checks out an application process from the pool and marks it as busy. If all app instances in the pool are busy and the pool limit has not been reached yet, then Passenger will spawn a new instance of the app…”

This means that Apache’s Concurrency level should at least be equal to the PassengerMaxPoolSize. Setting the web server’s concurrency higher than the maximum pool size wont give you additional concurrency. In simpler terms, if you have 20 Apache processes waiting to be served by 10 App Instances, then Apache may serve static assets like images and javascripts for processes 11 to 20 while waiting for the App Instances to be freed from the first 10 requests. On the other hand, limiting number of Apache processes to 10 when you have 20 Passenger App Instances to serve requests is under-usage of resources.

So once you have an average response time for your app from your ab tests, you can use the guideline below to determine your MaxClients.

(Avg. Response Time * (MaxClients – 1)) / (Apache’s concurrency) would be = Waiting time for last request

Again, use this only as a starting point. Load test various URLs in your application and see whether some actions consume more memory than others (for eg. a PDF rendering action might consume more memory owing to libraries that it uses).

Notes
Before getting to optimizing your server, I would strongly suggest that you find ways to optimize your application itself. Page caching, action caching, fragment caching and optimizing your DB queries are all great ways to improve application performance. Memcached and YSlow are tools you should definitely check out for caching. Consider using New Relic RPM to optimize your DB queries. For more on caching in Rails, try the following screencasts :

http://railslab.newrelic.com/scaling-rails
http://railscasts.com/tags/18

References
I owe a lot to the understanding gained from Hongli Lai from Phusion and the guys over at Slicehost Support (*Lee in particular). Both of these companies provide brilliant support and I’ve used a lot of their inputs in this series.

Here’re the resources I used :

Slow initial server startup when using Phusion Passenger and Rails
Holy Crap, My Rails Site is Slow
Ubuntu Intrepid – Apache configuration #1
Scaling and Performance: Apache2 memory usage too high; using swap. Help with configuraton.
Understanding Apache + Passenger directives – how do they work together?

Putting this together has been no small task. Yet I’m sure a lot of refinements can be made to the guidelines provided here. Feel free to leave your thoughts and suggestions below in the comments section.

Tags: , , , ,

Leave a comment