Page 1 of 2

[solved] python3 defunct processes with version 20.04

Posted: Thu Aug 06, 2020 11:00 am
by rodig03
I had to install python 3.8.5 for the upgrade.

Now I noticed 3 times (after less than 1 day) that I could not start a session as user openemm.
I found that the process limit was exhausted by defunct processes.
As the system is running idle, the number increases. Here after 15 minutes - 90 defunct processes

Code: Select all

[openemm@nl bin]$ ps -ef | grep python | grep 11017
openemm  11017 11016  0 11:43 ?        00:00:00 python3 /home/openemm/scripts/trigger3.py -bw [trigger3]
openemm  11816 11017  0 11:43 ?        00:00:00 [python3 /home/o] <defunct>
openemm  12783 11017  0 11:43 ?        00:00:00 [python3 /home/o] <defunct>
openemm  12843 11017  0 11:44 ?        00:00:00 [python3 /home/o] <defunct>
openemm  12870 11017  0 11:44 ?        00:00:00 [python3 /home/o] <defunct>
openemm  12872 11017  0 11:44 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13015 11017  0 11:44 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13019 11017  0 11:44 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13024 11017  0 11:45 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13027 11017  0 11:45 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13031 11017  0 11:45 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13034 11017  0 11:45 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13156 11017  0 11:45 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13158 11017  0 11:45 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13159 11017  0 11:46 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13165 11017  0 11:46 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13173 11017  0 11:46 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13174 11017  0 11:46 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13294 11017  0 11:46 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13298 11017  0 11:46 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13299 11017  0 11:47 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13300 11017  0 11:47 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13301 11017  0 11:47 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13306 11017  0 11:47 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13428 11017  0 11:47 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13430 11017  0 11:47 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13432 11017  0 11:48 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13436 11017  0 11:48 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13437 11017  0 11:48 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13440 11017  0 11:48 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13562 11017  0 11:48 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13564 11017  0 11:48 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13567 11017  0 11:49 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13568 11017  0 11:49 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13571 11017  0 11:49 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13578 11017  0 11:49 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13701 11017  0 11:49 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13703 11017  0 11:49 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13706 11017  0 11:49 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13709 11017  0 11:50 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13712 11017  0 11:50 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13713 11017  0 11:50 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13833 11017  0 11:50 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13835 11017  0 11:50 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13838 11017  0 11:50 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13841 11017  0 11:51 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13944 11017  0 11:51 ?        00:00:00 [python3 /home/o] <defunct>
openemm  14714 11017  0 11:51 ?        00:00:00 [python3 /home/o] <defunct>
openemm  14835 11017  0 11:51 ?        00:00:00 [python3 /home/o] <defunct>
openemm  14837 11017  0 11:51 ?        00:00:00 [python3 /home/o] <defunct>
openemm  14840 11017  0 11:51 ?        00:00:00 [python3 /home/o] <defunct>
openemm  14841 11017  0 11:52 ?        00:00:00 [python3 /home/o] <defunct>
openemm  14842 11017  0 11:52 ?        00:00:00 [python3 /home/o] <defunct>
openemm  14843 11017  0 11:52 ?        00:00:00 [python3 /home/o] <defunct>
openemm  14965 11017  0 11:52 ?        00:00:00 [python3 /home/o] <defunct>
openemm  14967 11017  0 11:52 ?        00:00:00 [python3 /home/o] <defunct>
openemm  14968 11017  0 11:52 ?        00:00:00 [python3 /home/o] <defunct>
openemm  14971 11017  0 11:53 ?        00:00:00 [python3 /home/o] <defunct>
openemm  14976 11017  0 11:53 ?        00:00:00 [python3 /home/o] <defunct>
openemm  14977 11017  0 11:53 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15103 11017  0 11:53 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15107 11017  0 11:53 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15112 11017  0 11:53 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15113 11017  0 11:54 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15116 11017  0 11:54 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15119 11017  0 11:54 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15239 11017  0 11:54 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15241 11017  0 11:54 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15246 11017  0 11:55 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15255 11017  0 11:55 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15260 11017  0 11:55 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15267 11017  0 11:55 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15389 11017  0 11:55 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15391 11017  0 11:55 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15394 11017  0 11:56 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15397 11017  0 11:56 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15398 11017  0 11:56 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15399 11017  0 11:56 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15523 11017  0 11:56 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15527 11017  0 11:56 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15528 11017  0 11:57 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15529 11017  0 11:57 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15530 11017  0 11:57 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15535 11017  0 11:57 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15657 11017  0 11:57 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15661 11017  0 11:57 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15662 11017  0 11:58 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15663 11017  0 11:58 ?        00:00:00 [python3 /home/o] <defunct>
[openemm@nl bin]$ ps -ef | grep python | grep 11017 | wc -l
90
It looks like I have to shut down / restart openemm as a workaround.

Is this a know issue? Can I give you any additional information?
Thanks + best regards,

Robert

Re: python3 defunct processes with version 20.04

Posted: Mon Aug 10, 2020 2:29 pm
by ud
Hello Robert,

unfortunately you encountered a bug and there is sadly no simple work around. We will fix the bug and provide an updated OpenEMM version. Thank you for reporting the bug and sorry for the trouble!

To keep the system running, you can at least restart the trigger process (as user openemm: "trigger.sh stop; trigger.sh start") on a regular base (e.g. every hour via cron), but this will not replace the proper bugfix, but should keep your process table clean.

-- ud

Re: python3 defunct processes with version 20.04

Posted: Tue Aug 11, 2020 5:17 pm
by rodig03
Hi ud,

thanks für that workaround.
Will there an impact on the frontend, sending mail, reacting on uer's clicks or webservices if trigger is down?
And ...
What are your plans for the release of the fix?

Robert

Re: python3 defunct processes with version 20.04

Posted: Tue Aug 11, 2020 6:10 pm
by maschoff
A fix for this bug (ID L-511) is already in testing. We expect a bugfix release for OpenEMM 20.04 next week.

Re: python3 defunct processes with version 20.04

Posted: Wed Aug 26, 2020 9:15 am
by rodig03
@maschoff
Do you have any news on the update? I just checked and I could not find any new program release.
Thanks!

Robert

Re: python3 defunct processes with version 20.04

Posted: Wed Aug 26, 2020 10:30 am
by maschoff
Backend update to 20.04.000.181 via OpenEMMInstaller schould work now.

Re: python3 defunct processes with version 20.04

Posted: Wed Aug 26, 2020 10:48 am
by rodig03
Thanks - now the update is installed. I am checking if everything is ok now.

Re: python3 defunct processes with version 20.04

Posted: Wed Aug 26, 2020 11:06 am
by maschoff
OK, at least, our internal tests were successful

Re: python3 defunct processes with version 20.04

Posted: Wed Aug 26, 2020 1:45 pm
by rodig03
Hi,

I have serious problems after the upgrade. Maybe this is a common reason, maybe there are different causes.
New: I cannot see the mailing preview even for mailings that were sent in the past.
Error in the preview windows

Code: Select all

Fehler in Dyn-Tags!
Textbaustein	agn-Tag
Fehler: Vorschau war leer	
Even it I create a new mailing with only "test" as a text body for text and html.

Second:
Since the update to 20.04 the automatic birthday mailings do not work.
Test sending worked, automaitic sending not. I played around with campaign manager and without - same result.
Pure target group evaluation about 500 users seems to work and give a different number each day.
Now these mails show the preview failure, too. I am not sure if they did have the preview error before the backend update, of if they do not work out of a different reason.

If you need any logs, please tell me where to look.
Thanks
Robert

Re: python3 defunct processes with version 20.04

Posted: Wed Aug 26, 2020 2:01 pm
by maschoff
Please update frontend to 20.04.000.185 and check again.

Re: python3 defunct processes with version 20.04

Posted: Wed Aug 26, 2020 2:03 pm
by ud
Could you please login as user openemm (on the server itself) and execute this command and paste the result as an answer?

ldd bin/xmlback

-- ud

Re: python3 defunct processes with version 20.04

Posted: Wed Aug 26, 2020 3:12 pm
by rodig03
Hi,
just updating to 20.04.000.185 did not change anything.
Seems some libraries are missing.

Code: Select all

[openemm@nl ~]$ ldd bin/xmlback
        linux-vdso.so.1 =>  (0x00007ffeeef48000)
        libsqlite3.so.0 => /lib64/libsqlite3.so.0 (0x00007f8d8467c000)
        libssl.so.1.1 => not found
        libcrypto.so.1.1 => not found
        libxml2.so.2 => /lib64/libxml2.so.2 (0x00007f8d84312000)
        libz.so.1 => /lib64/libz.so.1 (0x00007f8d840fc000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f8d83ef8000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f8d83bf6000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f8d83828000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f8d8360c000)
        liblzma.so.5 => /lib64/liblzma.so.5 (0x00007f8d833e6000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f8d84931000)
[openemm@nl ~]$

Re: python3 defunct processes with version 20.04

Posted: Thu Aug 27, 2020 12:52 pm
by ud
Hello,

which operation system in which version are you using, please?

-- ud

Re: python3 defunct processes with version 20.04

Posted: Thu Aug 27, 2020 4:04 pm
by rodig03
Hello,
this is centOS 7.
[root@nl ~]# cat /etc/centos-release
CentOS Linux release 7.8.2003 (Core)

[root@nl ~]# uname -a
Linux xxx 3.10.0-1127.18.2.el7.x86_64 #1 SMP Sun Jul 26 15:27:06 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Re: python3 defunct processes with version 20.04

Posted: Fri Aug 28, 2020 6:13 pm
by rodig03
Hi,
do I need to upgrade to CentOS 8 to get openssl 1.1 and get the system running?
Or is there a simpler way?
Seems you are relying on having version 1.1
Thanks

Robert