Memory Leak in WebServices

Use this forum to report bugs and to check for bugfixes and new releases of OpenEMM

Moderator: moderator

emmulator
Posts: 26
Joined: Mon Oct 27, 2008 7:44 pm

Memory Leak in WebServices

Post by emmulator »

I have been testing creating and sending Mailings using the webservices. One problem is that it takes 2 webservice calls for each subscriber I want to add (one to create and one to 'bind'). Then, while doing some load testing, I started getting OOM Exceptions, even though OpenEMM was running with xmx at 1.5 Gb. So I did a little profiling, and have noticed that after about 50k webservice calls (creating a mailing with 25k subscribers), the 'usedConnections' Hashtable managed by the WebServiceBase is holding on to 500Mb of RAM!

I imagine this Hashtable is maintained for some kind of webservice troubleshooting -- it appears to put a stacktrace in the Hashtable for each webservice call. But for some reason, the 'remove' is either not getting called or not working.

This probably wouldn't have shown up as a problem if there were an API that could be called to add more than one subscriber at a time. I was always dubious of using this double-ws call per subscriber, and now it looks like I'll be going around the API for this, as well as removing the usedConnections Hashtable itself, unless someone can demonstrate that it does something useful. :)
sp
Posts: 21
Joined: Mon Apr 12, 2010 2:04 pm

Confirming this

Post by sp »

We ran into the same issue. Is this recognized as a problem and if, is someone working on a fix?
emmulator
Posts: 26
Joined: Mon Oct 27, 2008 7:44 pm

Post by emmulator »

I don't know if it's been fixed in the official release, but I just removed the 'usedConnections' Hashtable and related code from the WebServiceBase, and it's been working fine ever since. We're still using version 5.5.1.
sp
Posts: 21
Joined: Mon Apr 12, 2010 2:04 pm

Post by sp »

Sadly it hasn't been fixed, we did the same with good results.
joostvb
Posts: 3
Joined: Wed Jan 23, 2008 11:36 am

Post by joostvb »

I'm using 6.0.1 and the 'java' process is running out of memory after making many calls to the webservice. It looks like the same problem as described above.
I patched the WebServiceBase.java file (removed all usage of usedConnections) and recompiled. With the new .class file in place, the memory usage of the 'java' process still consumes more memory after each call, but it's less then before.
maschoff
Site Admin
Posts: 2597
Joined: Thu Aug 03, 2006 10:20 am
Location: Munich, Germany
Contact:

Post by maschoff »

Since so many of you experienced this problem we will look into this issue and post our findings here.
OpenEMM Maintainer
mkalen
Posts: 33
Joined: Tue Jan 20, 2009 7:51 pm

Post by mkalen »

emmulator wrote:I don't know if it's been fixed in the official release, but I just removed the 'usedConnections' Hashtable and related code from the WebServiceBase, and it's been working fine ever since. We're still using version 5.5.1.
It seems the Hashtable "usedConnections" is a remnant of some sort of local connection pooling or connection tracking. Since AgnUtils.retrieveDataSource is now used to get a proper datasource from the servlet context (using Apache DBCP for connection pooling), the "usedConnections" have no use as far as I can see. (No references from code other than locally in WebServiceBase.java.)

I just ran into the same issue with out of memory after a lot of SOAP calls to OpenEMM (5.5.1 on CentOS) and profiled the server with YourKit Java Profiler. The results show that memory is filling up with string instances that are the values of the "usedConnections" hashtable (string created by the try/catch on rows 79-86).

As others in this thread, I could work around the problem by commenting out all code related to "usedConnections". Line numbers 52, 70-87 and 94 in the 5.5.1 version.
maschoff
Site Admin
Posts: 2597
Joined: Thu Aug 03, 2006 10:20 am
Location: Munich, Germany
Contact:

Post by maschoff »

The hashtable seems to be a leftover from a big debugging session and can be removed safely. We will fix that in the next release.
OpenEMM Maintainer
Post Reply