Tomcat Saturation

[mokarram]

Dear all,

When Openemm has more than a few hundred emails everything goes really sluggish. The server slows down to point where redirections take a few long seconds and finally timed out. This has happened a few times and I cannot figure out why. I tweaked the JVM memory to 2049-4096. The catalina log files has some info on:

Code: Select all

2012-02-27 15:50:25,187: ERROR [http-7881-3] org.agnitas - no url found for id:30
2012-02-27 15:51:54,932: INFO  [http-7881-2] org.agnitas.userlogs - admin: do load mailing Refresca su Imagen
2012-02-27 16:02:39,109: ERROR [http-7881-2] org.agnitas - no url found for id:27
202012-02-27 16:05:16,136: INFO  [http-7881-2] org.agnitas.userlogs - admin: do load mailing Refresca su Imagen
2012-02-27 16:32:18,818: ERROR [http-7881-6] org.agnitas - Http error connecting to 'http://www.openemm.org//version/2011/current_version.html'
2012-02-27 16:32:18,818: ERROR [http-7881-6] org.agnitas - The host did not accept the connection within timeout of 5000 ms
2012-02-27 16:38:06,143: ERROR [http-7881-5] org.agnitas - getDefaultValue: Can't find resource for bundle java.util.PropertyResourceBundle, key redirection.status
2012-02-27 16:38:06,143: ERROR [http-7881-5] org.agnitas - getDefaultValue: Can't find resource for bundle java.util.PropertyResourceBundle, key redirection.status2012-02-27 16:59:34,674: INFO  [http-7881-6] org.agnitas.userlogs - admin: do load mailing Refresca su Imagen
2012-02-27 17:00:07,835: ERROR [http-7881-7] org.agnitas - getDefaultValue: Can't find resource for bundle java.util.PropertyResourceBundle, key redirection.status

and lots of

Code: Select all

2012-02-27 15:02:49,486: ERROR [http-7881-3] org.agnitas - getDefaultValue: Can't find resource for bundle java.util.PropertyResourceBundle, key redirection.status

After sending 7,000 from a total of around 141,000 I recieved an error saying the the mails url were incorrect. Then it stopped on 7000.

Any help would be appreciated.

[unhuman]

Why are you hitting this url during normal processing?
http://www.openemm.org//version/2011/cu ... rsion.html

[mokarram]

Im not. I think its an internal process > http://www.openemm.org//version/2011/cu ... rsion.html

I have been playing around with jvm memory but as you said befre, it looks like another issue.

[maschoff]

The URL mentioned above is used to check at the login page if your OpenEMM is of the latest version. It should only be contacted at login time.

You should check your link with IDs 27 and 30.

To get rid of the error message regarding redirection.status you can set this value to 302 in file emm.properties.

[mokarram]

Hi,

Yes there were a couple of dead links in that mailing because I wanted to see what would happen if there is an external link in newsletter that changes. I dont think that is whats makes the server so sluggish


I deactivated the link checker to see if that would help in anyway and no change at all. exact same behavoir as before. seems like tomcat saturates and entire server looses external network connectivity. My newtork connection stays up and I can continue to use Internet however the Openemm server cannot see the outside world - hence not a bandwith issue. The behavoir has been consistent each time my masilings have been over a few hundred.

The SQL i found on the forum>

<code>INSERT INTO `admin_group_permission_tbl` (`admin_group_id`, `security_token`) VALUES (4, 'mailing.linkcheck.deactivate');</code>

This is at the beggining of a mailing with 12,000 recipients from my catalina.out in /home/openemm/logs

Code: Select all

2012-02-29 09:45:50,616: WARN  [http-7881-3] org.hibernate.util.JDBCExceptionReporter - SQL Error: 0, SQLState: null
2012-02-29 09:45:50,616: ERROR [http-7881-3] org.hibernate.util.JDBCExceptionReporter - Cannot create PoolableConnectionFactory (Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.)
2012-02-29 09:45:50,617: WARN  [http-7881-3] org.hibernate.util.JDBCExceptionReporter - SQL Error: 0, SQLState: null
2012-02-29 09:45:50,617: ERROR [http-7881-3] org.hibernate.util.JDBCExceptionReporter - Cannot create PoolableConnectionFactory (Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.)
2012-02-29 09:45:50,618: WARN  [http-7881-3] org.springframework.jdbc.support.SQLErrorCodesFactory - Error while extracting database product name - falling back to empty error codes
org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Communications link failure

and more...

Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.)
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:82)
at org.springframework.jdbc.support.JdbcUtils.extractDatabaseMetaData(JdbcUtils.java:280)
... 38 more
Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Communications link failure

Code: Select all

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.)
	at org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:855)
	at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:540)
	at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:113)
	at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:79)
	... 39 more
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

Aslo

Code: Select all

2012-02-29 10:19:00,947: ERROR [http-7881-1] org.agnitas - getDefaultValue: Can't find resource for bundle java.util.PropertyResourceBundle, key redirection.status

And finally when it just saturates and right before I shut down Openemm>

Code: Select all

SEVERE: The web application [/openemm] registered the JDBC driver [com.mysql.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
.
.
.
.
SEVERE: The web application [] appears to have started a thread named [MySQL Statement Cancellation Timer] but has failed to stop it. This is very likely to create a memory leak.
Feb 29, 2012 10:43:45 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [] appears to have started a thread named [pool-1-thread-1] but has failed to stop it. This is very likely to create a memory leak.

I think that there is a communication issues somewhere here, maybe a port that is bieng blocked that is needed by openemm? Why does it work when a small batch of recipients and goes nutty when sending large numbers? Are there ports that get used when number is high? Would anyone know which log file would be the best one to go through or is this pretty much a dead end?

Thanks