Memory Leaks in OpenEMM?

Use this forum to report bugs and to check for bugfixes and new releases of OpenEMM

Moderator: moderator

Shuro
Posts: 20
Joined: Mon Oct 26, 2009 1:26 pm
Location: Germany, Pr. Oldendorf

Memory Leaks in OpenEMM?

Post by Shuro » Fri Nov 05, 2010 9:45 am

We using OpenEMM two newsletters in a week, each has ~150.000 Recipients.
We use OpenEMM 6.1 and we use Web-services for Mail-Creation and Newsletter-Subscribers.

Multiple times we must restart OpenEMM cause Java-Errors in Statistics-Heatmap, which shows a Null-Pointer exception and in logs came this: "Cannot allocate memory".

OpenEMM runs under a Debian-Lenny Server with 3 GB Memory, 1,5 GB for OpenEMM and 1,5 GB for the System (sendmail & co).

The REAL problem:
xmlback breaks on sending emails! I noticed it on a Test-Mailing, but after some times of research, it also happened in productive Mailings! That cannot be tolerated, many mailings to recipients could be lost because that error.

And please, no solutions like "Increase the Memory on the Server", 3 GB is much enough, the problem is that java / openemm did not release the memory somewhere. I would profile it if i know how.


Here are some files:

/home/openemm/bin/core.sh

Code: Select all

export LANG=en_US.ISO8859_1
export RESIN_HOME=$HOME
$HOME/bin/httpd.sh -J-Xms1536m -J-Xmx1536m -conf $HOME/conf/core.conf -server-root $HOME/webapps/core -jvm-log $HOME/webapps/core/log/openemm_jvm.log -stderr $HOME/webapps/core/log/openemm_stderr.log -stdout $HOME/webapps/core/log/openemm_stdout.log -server core -pid $HOME/var/run/core.pid $args $*
/home/openemm/var/log/20101105-s4-mailgun.log

Code: Select all

[05.11.2010  08:49:14] ERROR/writer/meta/(1/2/103/125): command /home/openemm/bin/xmlback -vql -E /tmp/error5891881651176551113.tmp -o generate:temporary=true;syslog=false;account-logfile=/home/openemm/var/spool/log/account.log;bounce-logfile=log/extbounce.log;media=email;path=/home/openemm/var/spool/ADMIN /home/openemm/var/spool/META/AgnMail=D20101105084914=1=103=00001=liaMngA.xml failed (Missing binary? Wrong permissions?): java.io.IOException: Cannot run program "/home/openemm/bin/xmlback": java.io.IOException: error=12, Cannot allocate memory



/home/openemm/webapps/core/log/openemm_stderr.log

Code: Select all

[...]
triggerMailing: java.lang.Exception: Execution of /home/openemm/bin/xmlback -vql -E /tmp/error5891881651176551113.tmp -o generate:temporary=true;syslog=false;account-logfile=/home/openemm/var/spool/log/account.log;bounce-logfile=log/extbounce.log;media=email;path=/home/openemm/var/spool/ADMIN /home/openemm/var/spool/META/AgnMail=D20101105084914=1=103=00001=liaMngA.xml failed: java.io.IOException: Cannot run program "/home/openemm/bin/xmlback": java.io.IOException: error=12, Cannot allocate memory
org.agnitas.backend.MailWriterMeta.startXMLBack(MailWriterMeta.java:239)
org.agnitas.backend.MailWriterMeta.done(MailWriterMeta.java:289)
org.agnitas.backend.MailgunImpl.doExecute(MailgunImpl.java:583)
org.agnitas.backend.MailgunImpl.executeMailgun(MailgunImpl.java:116)
org.agnitas.beans.impl.MailingImpl.triggerMailing(MailingImpl.java:656)
org.agnitas.web.MailingSendAction.sendMailing(MailingSendAction.java:486)
org.agnitas.web.MailingSendAction.execute(MailingSendAction.java:218)
org.apache.struts.chain.commands.servlet.ExecuteAction.execute(ExecuteAction.java:58)
org.apache.struts.chain.commands.AbstractExecuteAction.execute(AbstractExecuteAction.java:67)
org.apache.struts.chain.commands.ActionCommandBase.execute(ActionCommandBase.java:51)
org.apache.commons.chain.impl.ChainBase.execute(ChainBase.java:190)
org.apache.commons.chain.generic.LookupCommand.execute(LookupCommand.java:304)
org.apache.commons.chain.impl.ChainBase.execute(ChainBase.java:190)
org.apache.struts.chain.ComposableRequestProcessor.process(ComposableRequestProcessor.java:283)
org.apache.struts.action.ActionServlet.process(ActionServlet.java:1913)
org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:449)
javax.servlet.http.HttpServlet.service(HttpServlet.java:115)
javax.servlet.http.HttpServlet.service(HttpServlet.java:92)
com.caucho.server.dispatch.ServletFilterChain.doFilter(ServletFilterChain.java:106)
org.springframework.orm.hibernate3.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:174)
[...]

Shuro
Posts: 20
Joined: Mon Oct 26, 2009 1:26 pm
Location: Germany, Pr. Oldendorf

Post by Shuro » Tue Nov 09, 2010 1:56 pm

Okay, it is a memory hole...

we have upgraded our openemm-server to 4 GB memory, 3 GB for openemm and 1 GB for system.

Today the same problem, test-Mailing wasn't generated because "Cannot allocate memory", only workaround was restarting openemm.

Here two Log-Files:

cat 20101109-s4-preview.log

Code: Select all

[09.11.2010  11:26:32] INFO/stat: Mailing cache: 0, Page cache: 0
[09.11.2010  11:26:32] INFO/create: Found error for 103/3: java.lang.Exception: Execution of /home/openemm/bin/xmlback -r -vql -E /tmp/error510625399793096045.tmp -o preview:path=/tmp/preview8265832474357819296.xml /home/openemm/var/spool/META/AgnMail=D20101109112632=1=103=00001=liaMngA.xml.gz failed: java.io.IOException: Cannot run program "/home/openemm/bin/xmlback": java.io.IOException: error=12, Cannot allocate memory
com.caucho.xml.XmlParseException: file:/tmp/preview8265832474357819296.xml:1: XML file has no top-element.  All well-formed XML files have a single top-level element.
cat 20101109-s4-mailgun.log

Code: Select all

[09.11.2010  11:26:32] ERROR/writer/meta/(1/2/103/-1): command /home/openemm/bin/xmlback -r -vql -E /tmp/error510625399793096045.tmp -o preview:path=/tmp/preview8265832474357819296.xml /home/openemm/var/spool/META/AgnMail=D20101109112632=1=103=00001=liaMngA.xml.gz failed (Missing binary? Wrong permissions?): java.io.IOException: Cannot run program "/home/openemm/bin/xmlback": java.io.IOException: error=12, Cannot allocate memory
[09.11.2010  13:41:06] ERROR/writer/meta/(1/1/104/179): command /home/openemm/bin/xmlback -vql -E /tmp/error144379447824201530.tmp -o generate:temporary=true;syslog=false;account-logfile=/home/openemm/var/spool/log/account.log;bounce-logfile=log/extbounce.log;media=email;path=/home/openemm/var/spool/ADMIN /home/openemm/var/spool/META/AgnMail=D20101109134106=1=104=00001=liaMngA.xml failed (Missing binary? Wrong permissions?): java.io.IOException: Cannot run program "/home/openemm/bin/xmlback": java.io.IOException: error=12, Cannot allocate memory

maschoff
Site Admin
Posts: 2177
Joined: Thu Aug 03, 2006 10:20 am
Location: Munich, Germany
Contact:

Post by maschoff » Wed Nov 24, 2010 5:54 pm

We investigated this problem, however, we did not use Debian.
The backend program "xmlback" should not be responsible for the problem because otherwise productive mailings would not cause the error in Java (sending of productive mailings does not involve Java).
Maybe the Java uses too much memory for the heap and the stack crashs? Or maybe the process list is full and no additional process can be started?
We use xmlback for really big mailings (up to 3.5 mio recipients) and it works.
However, we are in the process of writing a suite of JUnit tests right now (actually we are almost finished) and we identified some problems which could lead to a memory leak. As soon as we have new bugfixes or releases we will publish it at http://twitter.com/openemm .
OpenEMM Maintainer

sp
Posts: 21
Joined: Mon Apr 12, 2010 2:04 pm

Re: Memory Leaks in OpenEMM?

Post by sp » Mon Jan 17, 2011 4:37 pm

Calling exec from Java on some command will copy the whole memory of the application starting the command. If it's not possible, you will get the exception you have provided. For more information check http://stackoverflow.com/questions/2098 ... gards-to-m - it's not a memory leak, it's the way forking works on Linux.

Generally speaking, you want to use a lightweight process starter or to tweak your memory overcommit settings.

Daniell
Posts: 6
Joined: Thu Sep 08, 2011 1:45 pm

Re: Memory Leaks in OpenEMM?

Post by Daniell » Tue Sep 20, 2011 8:29 am

I need someone with the necessary experience with memory management in .Net to fix the memory leaks in the app. Thank you.

Post Reply