voyent
ICEFaces 1.7 potential memory leak  XML
Forum Index -> General Help
Author Message
dkohl

Joined: 02/Jun/2007 00:00:00
Messages: 13
Offline


Unfortunately 1.7 has an potential memory leak who makes it nearly unusable for production, even with higher memory settings.

Running the following page that only shows static text (and doesn't refer to beans) the memory decreases constantly.

Code:
 <?xml version="1.0" encoding="ISO-8859-1" ?>
 <jsp:root version="1.2" xmlns:jsp="http://java.sun.com/JSP/Page"
 	xmlns:f="http://java.sun.com/jsf/core"
 	xmlns:h="http://java.sun.com/jsf/html"
 	xmlns:ice="http://www.icesoft.com/icefaces/component">
 	<jsp:directive.page contentType="text/html;charset=ISO-8859-1"
 		pageEncoding="ISO-8859-1" />
 	<f:view>
 		<ice:outputDeclaration doctypeRoot="HTML"
 			doctypePublic="-//W3C//DTD HTML 4.01 Transitional//EN"
 			doctypeSystem="http://www.w3.org/TR/html4/loose.dtd" />
 		<html>
 			<head>
 				<title><ice:outputText value="ICEfaces, Ajax for Java EE" />
 				</title>
 				<ice:outputStyle href="./xmlhttp/css/royale/royale.css" />
 				
 				<ice:outputStyle href="style.css" />
 			</head>
 			<body>
 
 				<ice:form>
 					<ice:outputText value="Dummy"/>
 				</ice:form>
 
 			</body>
 		</html>
 	</f:view>
 </jsp:root>
 


i've profiled this session with (JProfiler 5.1.3) and called it with JMeter. Session timeout 5 minutes. garbage collection doesn't free memory too.

any ideas?
[Thumb - ps_heap.JPG]
 Filename ps_heap.JPG [Disk] Download
 Description JProfiler View Heap
 Filesize 98 Kbytes
 Downloaded:  416 time(s)

[Thumb - ps_heap.JPG]
 Filename ps_heap.JPG [Disk] Download
 Description JProfiler View PS Old Gen
 Filesize 98 Kbytes
 Downloaded:  381 time(s)

greg.dick

Joined: 26/Oct/2004 00:00:00
Messages: 85
Offline


I've created a page with this jsp, and don't find a problem, however, there is some behaviour that isn't immediately obvious when using JMeter and/or other automated load generating tools.

In an asynchronous ICEfaces application (defined by:

<context-param>
<param-name>com.icefaces.faces.synchronousUpdate</param-name>
<param-value>true/false</param-value>
</context-param>

the javascript bridge is informed of session expiry by the server and it then submits a POST request of the form:

${app}/block/dispose-views

to release the resources for the views and session. JMeter doesn't perform this protocol, by default, and the result is a 15 second timeout penalty (per session) as the server waits for the protocol to complete. I used 100 clients against my test app and barely went from 10Meg Heap to just slightly over 20Meg and that takes 60 minutes for all the sessions to expire and for the memory to be reclaimed and from the size of your heap allocation I'd guess you have a whole lot more clients than that.

I'm attaching a sample JMeter script that I've worked out that should perform the session expiry protocol properly. This script has some problems too. Notably, it has a hard time scaling over just a few tens of users for reasons unknown. But it does highlight some of the difficulties in testing the ICEfaces asynchronous connections through JMeter.


Some pseudocode for the protocol follows:


while (session != expired)

browser->server: ${app}/block/receive-updated-views
- this request blocks at the server until the server has updates for the session, either server push, or generated by a PING request (below). This is followed by:

browser->server: ${app}/block/receive-updates (for View)
- recieve updates for the view specified in the receive-updated-views response.

end-while

If there is no response for 50 seconds (configurable via context-param) the browser sends:

browser->server: ${app}/block/ping

which sends traffic to keep network connections alive and releases the blocked receive-updated-views request and returns a <PONG/> response from the subsequent receive-updates request.


The nature of the blocking request makes it difficult to generate traffic at a particular rate in JMeter effectively. It's very easy to wind up with all the Threads in a threadGroup blocking, with no one generating any traffic at all. In JMeter the user variables are not shared between threadGroups, so it's difficult to have an independent threadGroup submit PING with the correct ice.session and ice.view parameters.


Anyway, I would suggest trying your script with 30 or so users in an effort to minimize the amount of time it takes for all the sessions to serially expire and check then to see if your memory has been recovered.

 Filename simpleAppJSPExpiry.jmx [Disk] Download
 Description JMeter script containing session-expiry protocol.
 Filesize 29 Kbytes
 Downloaded:  515 time(s)

[Email]
dkohl

Joined: 02/Jun/2007 00:00:00
Messages: 13
Offline


hi greg,

thanks for your really detailed answer!

i checked my site with 1.6.2 and exactly the same source, and there wasn't a memory leak at all, running 60.000 (in words sixty-thousand) sessions called by a simple JMeter script over the weekend. no leaks at all! no "out of memory"-exceptions! it runs really fine! and my app is a really big one! :-)

changing the build path to 1.7 libs, running under exactly the same conditions and infrastructure, results in "out-of-memory" error after a few hundred sessions.

i think this is a little bit strange an has the flavour of a bug :-)

the "call"-conditions outside should not be responsible for memory-leaks inside.


greetings

daniel

greg.dick

Joined: 26/Oct/2004 00:00:00
Messages: 85
Offline


Ok, can you give us a bit more information about your application? Such as:

Asynchronous or synchronous.
Are you using any server push rendering groups.
Request or sessions scoped beans.
Standard or extended request scope.
Any other extensions or third party frameworks.

Have you updated the regular expressions in JMeter to extract the ice.view and ice.session parameters because they've changed between 1.6 and 1.7?

And perhaps some info on the nature of your Jmeter scripts. Do they operate in a loop with a fixed set of Sessions, or continuously allocate and discard them, etc.

This info would be helpful.

Greg
[Email]
dkohl

Joined: 02/Jun/2007 00:00:00
Messages: 13
Offline


greg.dick wrote:
Ok, can you give us a bit more information about your application? Such as:

Asynchronous or synchronous.
 

i checked both modes as desrcibed in other comparable postings. the mode doesn't matter in my case. Concurrent DOMs didn't solve it too. it's only version dependend (1.6.2 or 1.7)

greg.dick wrote:

Are you using any server push rendering groups.
 

no server initiated rendering.


greg.dick wrote:

Request or sessions scoped beans.
 

ONLY session beans. i never use request beans.
two application-scoped beans used.

greg.dick wrote:

Standard or extended request scope.
 

i've not changed the standard settings. so, beans are being killed after session timeout (i also tried different times from 1min - 5min).


greg.dick wrote:

Any other extensions or third party frameworks.
 

Hibernate 3.2 and its dependencies and mysql, thats it (i can you send my lib-dir in detail if you want). java version doesnt matter.

greg.dick wrote:

Have you updated the regular expressions in JMeter to extract the ice.view and ice.session parameters because they've changed between 1.6 and 1.7?
 

i only use the standards. i added a endless-thread group, a constant timer (from 20ms up to 2000ms), and an unspectacular standard-HTTP-sampler :-)


greg.dick wrote:

And perhaps some info on the nature of your Jmeter scripts. Do they operate in a loop with a fixed set of Sessions, or continuously allocate and discard them, etc.
 

sure , i can send it to you tomorrow. but it's just created with a normal JMeter described the way above. nothing more. simple calls! nothing special.


greg.dick wrote:
This info would be helpful.
 

i can also give some more details about your source from JProfiler, but i need a bit time to set it up.


greetings

schroederw

Joined: 19/Nov/2007 00:00:00
Messages: 12
Offline


Is it possible, that the problem is (partly) related to http://www.icefaces.org/JForum/posts/list/8482.page ?

At least, this should cause a small memeory leak, because due tue the exception the session-ids are not removed from the list anymore. What I can't tell is, if the session-shutdown-notify may be called from elsewhere but session.invalidate(). If this is the case, the code after that call is interruppted as well and potential cleanup routines are not run anymore.
dkohl

Joined: 02/Jun/2007 00:00:00
Messages: 13
Offline


schroederw wrote:
Is it possible, that the problem is (partly) related to http://www.icefaces.org/JForum/posts/list/8482.page ?
 

i've never seen this exception. my sessions are never invalidated manually.
and the memory leak is much bigger than you would expect.
with 256MB reserved under Tomcat 6 the memory will run down to about 50MB after 2000 sessions and will never be freed by the GC using 1.7 and the conditions mentioned above.

greetings

daniel
greg.dick

Joined: 26/Oct/2004 00:00:00
Messages: 85
Offline


Hi,

The problem is as I posted above. It's a very simply app and test script which makes new requests and creates new sessions at a rate of a new session every 200 ms or thereabouts.

The problem is that when the sessions expire after 5 minutes, the server inserts a <session-expired> response into the command queue of the user, and waits up to 15 seconds to receive the /block/dispose-views command (for all views associated with the session). Generally, in a browser environment, the interaction is rapid and the sessions are disposed of rapidly.

The 1.6 implementation found that browsers could continue to make asynchronous requests on a freshly expired session, before it processed the <session-expiry> response from the server, thereby creating new sessions in error.

I see that you are using synchronous connections (ie. no blocking connections, no AJAX interaction at all), another problem with the current implementation is that the server is still waiting for the response to <session-expired> despite there being no agent in the browser in this case to respond to it. In this case, the maximum wait per session is unavoidable.


This is new development between 1.7 and 1.6, which is why it shows up now.

I've created A JIRA to track this issue:
http://jira.icefaces.org/browse/ICE-3105
[Email]
dkohl

Joined: 02/Jun/2007 00:00:00
Messages: 13
Offline


greg.dick wrote:
Hi,
...
I see that you are using synchronous connections (ie. no blocking connections, no AJAX interaction at all)
...
 

it's independent of this setting. i checked also the concurrentDOMview-setting true or false (as suggested in other postings). doesn't solve the problem, too.

greetings

daniel
maratb

Joined: 17/Dec/2006 00:00:00
Messages: 13
Offline


Just to raise awarness of how bad memory is being leacked by IceFaces 1.7.1 with Facelets.

Have a look at this Jira:

http://jira.icefaces.org/browse/ICE-3231

Basically as far as I can see most of leaks come down to the fact that the server creates more than 1 server Views for the same UI View. This leads to MainSessionBoundServlet.views holding stale Views as they aren't removed when the client disposes of a single view that it only knows about, others remain in this map I suppose until server side session expires.

But because facelets are used a single View can take up megabytes of memory and you don't need a lot of sessions to have JVM bomb (OutOfMemory exceptions)
ken.fyten

Joined: 26/Oct/2004 00:00:00
Messages: 1337
Offline


Marat,

As the issue you are currently reporting is not related to the previous issues on this thread (that pre-date 1.7.1), please create a new thread for it.

Also, I noticed you've added some comments to ICE-3231. Note that ICE-3231 documents a specific issue with Facelets (actually a Facelets issue, not an ICEfaces specific one) that is different from the issue you are experiencing. Please create an new JIRA for your issue once you have a suitable test-case that reproduces it succinctly, etc. In the meantime I'm going to remove your comments from ICE-3231 to keep it focused on the issue it represents to avoid possible confusion, etc.

Regards,
Ken

Ken Fyten
VP Product Development
ICEsoft Technologies, Inc.
 
Forum Index -> General Help
Go to:   
Powered by JForum 2.1.7ice © JForum Team