How will JSF 2 behave in terms of performance for my web application? What are the trade-offs? How does it work compared with other web frameworks? These are difficult questions to answer. Performance is important in a web framework, even though it is certainly not the only consideration. But at least it's good to have a general idea about how JSF 2.0 performs. In the last couple years (2011-2012), this topic has received a lot of attention in the MyFaces Community, and a lot of improvements have been made, aiming to provide an outstanding JSF implementation.

To see how effective the improvements are, I compared JSF 2 with a similar Java Web Framework, Apache Wicket. In order to save some work, I used the sample booking web application bundled originally in the Seam 2.2.x framework, which has also been used in previous comparisons. The application has been upgraded to JSF 2 and Wicket 1.5, so we'll compare JSF against two Wicket versions (1.4.20 and 1.5.5). Since the intention is to look at how JSF works from a performance perspective, Wicket will serve only as a reference point. Before we set up the application, let's look at what we mean by performance.

What is Performance?

In this case, performance is about how the web framework uses the available resources to do useful work. Each framework tries to achieve a balance between different aspects. A good benchmark should consider those aspects and how they are related. So for this comparison, the following aspects will be checked:

  • Speed: Check the speed of the code under different situations. If the framework is fast it will process more requests with less CPU consumption.
  • Memory Consumption: Check how much memory the application needs to allocate for a specific task, and how the application behaves under low memory conditions. If the framework requires less memory, less time will be used in garbage collection and the framework will perform better under stress conditions.
  • Session Size: Check how much memory is used to store the session. If it uses less memory, it will be able to serve more users with the same memory size, and if the session is stored somewhere (database, file, memory cache... ), less time will be used in processing that information (usually serialization/deserialization and dealing with I/O operations).

Sample Application

The test project is a simple hotel booking application example provided by Seam 2.2.x. The example was initially chosen because it provides some typical use cases, without being too complex to understand. It provides a login page, a view with some data tables, a wizard to create a booking, an option to logout, some Ajax cases, and some forms that require validation. So without being too extensive, the example uses the most relevant parts of a web framework like JSF or Apache Wicket.

The operations considered in the test are:

  • Get login: render the login page.
  • Post login: the user introduces his or her username/password and clicks on the button; the server answers with an empty page.
  • Ajax post search: the user clicks on an Ajax button and through Ajax, a data table is filled with 10 registers.
  • Get view hotel: the user clicks on a link, choosing a hotel to make a reservation. A page with information about the hotel is rendered.
  • Post book hotel: the user confirms he/she wants to make a reservation by clicking on a button and then a form is displayed with the necessary information.
  • Ajax post CC number: the user type a credit card number and an AJAX request is sent to the server to confirm whether the entered number is valid or not.
  • Ajax post CC name: the user types a credit card name and an Ajax request is sent to the server to confirm if the name is valid or not.
  • Post booking details: the user clicks on the "Proceed" button, and a confirmation page is displayed with the submitted information.
  • Post confirm booking: the user clicks on the "Confirm" button, and the page with the hotels available and the booking reservation is rendered, with a message that says the reservation has been registered.
  • Get cancel booking: the user cancels the reservation, clicking on a link.
  • Logout: the user logs out from the system and the login page is presented.

There was an initial comparison done some years ago, running this application first in JSF 1.2 plus Seam and then in Wicket 1.4. To make this comparison more accurate, this time the base code was rewritten to use JSF 2.0 and JPA only, in the same way the example using Wicket was done. In this way, the results are comparable because the business logic and data access layer do more or less the same things.

Install and Run the Sample Application

The test environment used for this benchmark looks like this:


Figure 1. Deployment diagram

On one side we have JMeter running in a computer, and on the other side there is a Tomcat server running Hibernate with an in-memory database (hsqldb). On top of that, there is the web framework, which can be MyFaces, Mojarra, or Wicket. You can download the zip file containing the files related to this comparison from here.

The version used in this comparison is 2.0. You'll see this structure:

  • /perfbench-code: Booking application using different frameworks.
  • /comparison-2012-APR: Experimental data used in this comparison.
  • /state-size-test: Test to check how the JSF Partial State Saving algorithm works—just type mvn clean jetty:run to run it.

In /perfbench-code you can find:

  • /myfaces-jpa: JSF 2.0 booking application using MyFaces (current version 2.1.7)
  • /mojarra-jpa: JSF 2.0 booking application using Mojarra (current version 2.1.7)
  • /wicket-jpa: Wicket 1.4 booking application
  • /wicket15-jpa: Wicket 1.5 booking application

You can also find the test application source code and the JMeter scripts in the zip file.

Install and Run Tomcat Server

Setting up Tomcat for the comparison is quite simple; just follow these steps:

  1. Download and install Tomcat 7.0.x
  2. Inside conf/server.xml file, set autoDeploy="false"

       <Host name="localhost"  appBase="webapps"

            unpackWARs="true" autoDeploy="false">

 

  1. Inside conf/server.xml file, remove or comment AccessLogValve entry

        <!--  MyFaces PerfBench: Remove AccessLogValve

        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"

               prefix="localhost_access_log." suffix=".txt"

               pattern="%h %l %u %t &quot;%r&quot; %s %b" />

         -->

 

  1. Use JAVA_HOME, CATALINA_OPTS or whatever to set the server params. For example, in Windows:

 

SET JAVA_HOME=C:\jdk1.6.0_30

SET JAVA_OPTS=-Xms128m -Xmx128m -server

 

In Linux or Mac OS X:

 

export JAVA_HOME=/usr/to/your/java/jdk1.6.0_30

export JAVA_OPTS="-Xms128m -Xmx128m -server"

 

You can just modify catalina.bat or catalina.sh and add the required lines at the beginning of the script.

 

  1. (Optional) Set up Tomcat to use your selected profiler (for example YourKit or Netbeans Profiler).

 

After these steps, it is a good idea to copy the modified Tomcat installation, and reuse it for each WAR you try.

There are zip files for each test app configuration in the comparison-2012-APR/webapps folder of the zip file, or you can download them here.  Just download the required .zip file and uncompress it in the tomcat/webapps folder. Start Tomcat as usual and point to the installed web application.

Install and Run JMeter

An Ant script is used to run the JMeter scripts located in the src/test/jmeter folder of each test application. For example, in MyFaces, look in /myfaces-jpa/src/test/jmeter folder. You can also run JMeter without Ant just by opening the file booking-jmeter-direct.jmx, but remember to set up the parameters before you run any test. Before you run the scripts using Ant, do the following steps:

Install Ant and make sure the ant command can be found from the command line.

Create a build.properties file and update the params so the script can find the JMeter directory and knows where the web server is located (and running!).

After that you can go to <your selected test app>-jpa (for example myfaces-jpa), and type:

   ant jmeter-custom

Enter the required parameters and that's it. To warmup the server, type:

   ant jmeter-custom-warmup

Try the sample application to see how it works, take a look at the source code, and run some tests with JMeter. Next, we'll use the sample application to analyze JSF 2.0 from the speed perspective.

Speed Benchmark

The important point to take into account here—more than knowing the speed with which JSF is capable of handling multiple concurrent requests—is to understand the relationship between how much useful work the framework is able to do and how much time is spent in doing it. Since JSF and Wicket are different frameworks, they have different solutions to the same problem, and the results should be analyzed taking that into account.

Also, there is a balance between what the framework does for you and what you have to do when you build your custom web application. Including more assistive code into a framework means more lines of code need to be executed, which has an impact on speed. On the other hand, more assistive code means lower maintenance costs.

Step 1: Code Speed Under Minimal Load

The first step is to take a look at how each framework performs under minimum load. The information gathered here is useful in identifying bottlenecks in code speed, but does not take into account other effects like concurrency. Based on the previous considerations, the following experiments were done.

CONFIGURATION

Processor: AMD Phenom x4

Speed:     2300 MHz

Server: Apache Tomcat 7.0.26

JVM version : oracle jdk1.6.0_30

JVM Options : -Xms128m -Xmx128m -server

WARMUP

loop.count: 1500

thread.count: 5

include.logout : 1

include.delete: 1

booking.count: 1

include.ajax: 1

thread.delay: 10                           

thread.deviation: 5

rampup time: 1s

Http Request Client : Httpclient4

EXPERIMENT

loop.count: 100

thread.count: 5

include.logout : 1

include.delete: 1

booking.count: 1

include.ajax: 1

thread.delay: 0

thread.deviation:0

rampup time: 1s

Http Request Client : Java

 

Basically, we have a 4 processor server with only 5 threads running on the server side, sequentially executing each operation (get login, post login, ….. get cancel booking, get logout). See the graph below:

 


Figure 2. 90% line with 5 threads for Booking Application

Note that Wicket behaves differently when disk page storage is used (default) than when http session page storage is used, as any operation related to the disk will be slower than an operation done in memory. In some cases it is better to stick with the default disk mode for Wicket and keep page storage outside the session, which will reduce the session size per user, at the cost of some additional CPU consumption and I/O operations. However, when comparing Wicket to JSF, it is better to use session storage because in that mode it is possible to see Wicket without that overhead (but remember that using session storage has a cost because it makes the state size bigger).

Step 2: Code Speed with Load

Now let's see what happens if we run the tests using more threads, while keeping enough memory resources. This experiment makes it easier to see "how much contention overhead" a web framework has, and the results will be closer to the reality. These are the experiment parameters:

 

CONFIGURATION

Processor: AMD Phenom x4

Speed:     2300 MHz

Server: Apache Tomcat 7.0.26

JVM version : oracle jdk1.6.0_30

JVM Options : -Xms128m -Xmx128m -server

WARMUP

loop.count: 200

thread.count: 40

include.logout : 1

include.delete: 1

booking.count: 1

include.ajax: 1

thread.delay: 20                     

thread.deviation:10

rampup time: 1s

Http Request Client : Httpclient4

EXPERIMENT

loop.count: 100

thread.count: 40

include.logout : 1

include.delete: 1

booking.count: 1

include.ajax: 1

thread.delay: 0

thread.deviation:0

rampup time: 5s

Http Request Client : Java


Note the rampup time on the experiment was increased to 5 seconds, because there are more threads to create. These are the results:


Figure 3. 90% line with 40 threads for Booking Application

Here Wicket 1.4.20 is the fastest one, but note how slow the "post book hotel" is compared with JSF, which means even using session storage, there is a significant delay in that part.

Step 3: Code Speed with Increasing Load

In order to validate the results gathered in step 2, it is necessary to check the behavior when the load is increasing, varying the number of threads. Some changes were made, such as increasing the memory to 512m, using Httpclient4 as the http request client instead of Java, and providing a longer warmup time. Here are the experiment parameters:

CONFIGURATION

Processor: AMD Phenom x4

Speed:     2300 MHz

Server: Apache Tomcat 7.0.26

JVM version : oracle jdk1.6.0_16 for linux

JVM Options : -Xms512m -Xmx512m -server

WARMUP

loop.count: 200

thread.count: 100

include.logout : 1

include.delete: 1

booking.count: 1

include.ajax: 1

thread.delay: 20                           

thread.deviation:10

rampup time: 20s

Http Request Client : Httpclient4

EXPERIMENT

loop.count: 100

thread.count: 10, 20, 30, 40, 60, 80, 120, 160, 200

include.logout : 1

include.delete: 1

booking.count: 1

include.ajax: 1

thread.delay: 0

thread.deviation:0

rampup time: 1, 2, 3, 4, 6, 8, 12, 16, 20s

Http Request Client : Httpclient4

 

The changes made do not lead to any significant difference. Here is the resulting graph:


Figure 4. 90% line (Response Time) vs Thread Number for Booking Application 100 loop and 512m

The graph above is important for multiple reasons. First of all, you can see the difference between Wicket 1.4.20 and 1.5.5. In this case, a much better programming model usually means lower response times. Also, remember that Wicket uses disk storage by default, so it is not surprising that in some cases Wicket can perform slower than Mojarra, but that will depend on the underlying hardware.

In JSF 2, instead of storing the whole page somewhere, an algorithm was used to overcome this limitation called Partial State Saving (PSS). Later we'll see the impact in the state, but what's important to take into account here is that the response time of MyFaces 2.1.7 is similar to Wicket 1.5.5, which suggest the overhead involved in PSS is minimal compared with the overhead involved in Wicket disk storage.

The graph also suggests that the JSF lifecycle does not impose significant overhead. The JSF specification has been carefully designed to have a good balance between the functionality offered and performance considerations. In practice, the impact of the different JSF phases is small, compared with the time spent rendering the view, building the component tree, or executing the validation logic. In this case, a framework like Wicket does more or less the same steps, so there is no fundamental reason to think that JSF cannot have good response times in this part. The evidence is just overwhelming.

Next Step...

In the next article, we'll see how memory behaves under different conditions, and how it is related to CPU consumption. Then we'll look at how the state per view and session size behave, and the benefit of use JSF 2.0 Partial State Saving.