[solved] python3 defunct processes with version 20.04

Use this forum for questions regarding installation, upgrading and configuration of OpenEMM

Moderator: moderator

rodig03
Posts: 44
Joined: Thu Sep 18, 2014 1:27 pm

[solved] python3 defunct processes with version 20.04

Post by rodig03 » Thu Aug 06, 2020 11:00 am

I had to install python 3.8.5 for the upgrade.

Now I noticed 3 times (after less than 1 day) that I could not start a session as user openemm.
I found that the process limit was exhausted by defunct processes.
As the system is running idle, the number increases. Here after 15 minutes - 90 defunct processes

Code: Select all

[openemm@nl bin]$ ps -ef | grep python | grep 11017
openemm  11017 11016  0 11:43 ?        00:00:00 python3 /home/openemm/scripts/trigger3.py -bw [trigger3]
openemm  11816 11017  0 11:43 ?        00:00:00 [python3 /home/o] <defunct>
openemm  12783 11017  0 11:43 ?        00:00:00 [python3 /home/o] <defunct>
openemm  12843 11017  0 11:44 ?        00:00:00 [python3 /home/o] <defunct>
openemm  12870 11017  0 11:44 ?        00:00:00 [python3 /home/o] <defunct>
openemm  12872 11017  0 11:44 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13015 11017  0 11:44 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13019 11017  0 11:44 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13024 11017  0 11:45 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13027 11017  0 11:45 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13031 11017  0 11:45 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13034 11017  0 11:45 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13156 11017  0 11:45 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13158 11017  0 11:45 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13159 11017  0 11:46 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13165 11017  0 11:46 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13173 11017  0 11:46 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13174 11017  0 11:46 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13294 11017  0 11:46 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13298 11017  0 11:46 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13299 11017  0 11:47 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13300 11017  0 11:47 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13301 11017  0 11:47 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13306 11017  0 11:47 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13428 11017  0 11:47 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13430 11017  0 11:47 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13432 11017  0 11:48 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13436 11017  0 11:48 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13437 11017  0 11:48 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13440 11017  0 11:48 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13562 11017  0 11:48 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13564 11017  0 11:48 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13567 11017  0 11:49 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13568 11017  0 11:49 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13571 11017  0 11:49 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13578 11017  0 11:49 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13701 11017  0 11:49 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13703 11017  0 11:49 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13706 11017  0 11:49 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13709 11017  0 11:50 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13712 11017  0 11:50 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13713 11017  0 11:50 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13833 11017  0 11:50 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13835 11017  0 11:50 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13838 11017  0 11:50 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13841 11017  0 11:51 ?        00:00:00 [python3 /home/o] <defunct>
openemm  13944 11017  0 11:51 ?        00:00:00 [python3 /home/o] <defunct>
openemm  14714 11017  0 11:51 ?        00:00:00 [python3 /home/o] <defunct>
openemm  14835 11017  0 11:51 ?        00:00:00 [python3 /home/o] <defunct>
openemm  14837 11017  0 11:51 ?        00:00:00 [python3 /home/o] <defunct>
openemm  14840 11017  0 11:51 ?        00:00:00 [python3 /home/o] <defunct>
openemm  14841 11017  0 11:52 ?        00:00:00 [python3 /home/o] <defunct>
openemm  14842 11017  0 11:52 ?        00:00:00 [python3 /home/o] <defunct>
openemm  14843 11017  0 11:52 ?        00:00:00 [python3 /home/o] <defunct>
openemm  14965 11017  0 11:52 ?        00:00:00 [python3 /home/o] <defunct>
openemm  14967 11017  0 11:52 ?        00:00:00 [python3 /home/o] <defunct>
openemm  14968 11017  0 11:52 ?        00:00:00 [python3 /home/o] <defunct>
openemm  14971 11017  0 11:53 ?        00:00:00 [python3 /home/o] <defunct>
openemm  14976 11017  0 11:53 ?        00:00:00 [python3 /home/o] <defunct>
openemm  14977 11017  0 11:53 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15103 11017  0 11:53 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15107 11017  0 11:53 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15112 11017  0 11:53 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15113 11017  0 11:54 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15116 11017  0 11:54 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15119 11017  0 11:54 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15239 11017  0 11:54 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15241 11017  0 11:54 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15246 11017  0 11:55 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15255 11017  0 11:55 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15260 11017  0 11:55 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15267 11017  0 11:55 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15389 11017  0 11:55 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15391 11017  0 11:55 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15394 11017  0 11:56 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15397 11017  0 11:56 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15398 11017  0 11:56 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15399 11017  0 11:56 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15523 11017  0 11:56 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15527 11017  0 11:56 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15528 11017  0 11:57 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15529 11017  0 11:57 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15530 11017  0 11:57 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15535 11017  0 11:57 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15657 11017  0 11:57 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15661 11017  0 11:57 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15662 11017  0 11:58 ?        00:00:00 [python3 /home/o] <defunct>
openemm  15663 11017  0 11:58 ?        00:00:00 [python3 /home/o] <defunct>
[openemm@nl bin]$ ps -ef | grep python | grep 11017 | wc -l
90
It looks like I have to shut down / restart openemm as a workaround.

Is this a know issue? Can I give you any additional information?
Thanks + best regards,

Robert
Last edited by rodig03 on Fri Sep 04, 2020 10:07 am, edited 1 time in total.

ud
Posts: 153
Joined: Thu Aug 17, 2006 9:56 am
Location: Munich, Germany
Contact:

Re: python3 defunct processes with version 20.04

Post by ud » Mon Aug 10, 2020 2:29 pm

Hello Robert,

unfortunately you encountered a bug and there is sadly no simple work around. We will fix the bug and provide an updated OpenEMM version. Thank you for reporting the bug and sorry for the trouble!

To keep the system running, you can at least restart the trigger process (as user openemm: "trigger.sh stop; trigger.sh start") on a regular base (e.g. every hour via cron), but this will not replace the proper bugfix, but should keep your process table clean.

-- ud

rodig03
Posts: 44
Joined: Thu Sep 18, 2014 1:27 pm

Re: python3 defunct processes with version 20.04

Post by rodig03 » Tue Aug 11, 2020 5:17 pm

Hi ud,

thanks für that workaround.
Will there an impact on the frontend, sending mail, reacting on uer's clicks or webservices if trigger is down?
And ...
What are your plans for the release of the fix?

Robert

maschoff
Site Admin
Posts: 2493
Joined: Thu Aug 03, 2006 10:20 am
Location: Munich, Germany
Contact:

Re: python3 defunct processes with version 20.04

Post by maschoff » Tue Aug 11, 2020 6:10 pm

A fix for this bug (ID L-511) is already in testing. We expect a bugfix release for OpenEMM 20.04 next week.
OpenEMM Maintainer

rodig03
Posts: 44
Joined: Thu Sep 18, 2014 1:27 pm

Re: python3 defunct processes with version 20.04

Post by rodig03 » Wed Aug 26, 2020 9:15 am

@maschoff
Do you have any news on the update? I just checked and I could not find any new program release.
Thanks!

Robert

maschoff
Site Admin
Posts: 2493
Joined: Thu Aug 03, 2006 10:20 am
Location: Munich, Germany
Contact:

Re: python3 defunct processes with version 20.04

Post by maschoff » Wed Aug 26, 2020 10:30 am

Backend update to 20.04.000.181 via OpenEMMInstaller schould work now.
OpenEMM Maintainer

rodig03
Posts: 44
Joined: Thu Sep 18, 2014 1:27 pm

Re: python3 defunct processes with version 20.04

Post by rodig03 » Wed Aug 26, 2020 10:48 am

Thanks - now the update is installed. I am checking if everything is ok now.

maschoff
Site Admin
Posts: 2493
Joined: Thu Aug 03, 2006 10:20 am
Location: Munich, Germany
Contact:

Re: python3 defunct processes with version 20.04

Post by maschoff » Wed Aug 26, 2020 11:06 am

OK, at least, our internal tests were successful
OpenEMM Maintainer

rodig03
Posts: 44
Joined: Thu Sep 18, 2014 1:27 pm

Re: python3 defunct processes with version 20.04

Post by rodig03 » Wed Aug 26, 2020 1:45 pm

Hi,

I have serious problems after the upgrade. Maybe this is a common reason, maybe there are different causes.
New: I cannot see the mailing preview even for mailings that were sent in the past.
Error in the preview windows

Code: Select all

Fehler in Dyn-Tags!
Textbaustein	agn-Tag
Fehler: Vorschau war leer	
Even it I create a new mailing with only "test" as a text body for text and html.

Second:
Since the update to 20.04 the automatic birthday mailings do not work.
Test sending worked, automaitic sending not. I played around with campaign manager and without - same result.
Pure target group evaluation about 500 users seems to work and give a different number each day.
Now these mails show the preview failure, too. I am not sure if they did have the preview error before the backend update, of if they do not work out of a different reason.

If you need any logs, please tell me where to look.
Thanks
Robert

maschoff
Site Admin
Posts: 2493
Joined: Thu Aug 03, 2006 10:20 am
Location: Munich, Germany
Contact:

Re: python3 defunct processes with version 20.04

Post by maschoff » Wed Aug 26, 2020 2:01 pm

Please update frontend to 20.04.000.185 and check again.
OpenEMM Maintainer

ud
Posts: 153
Joined: Thu Aug 17, 2006 9:56 am
Location: Munich, Germany
Contact:

Re: python3 defunct processes with version 20.04

Post by ud » Wed Aug 26, 2020 2:03 pm

Could you please login as user openemm (on the server itself) and execute this command and paste the result as an answer?

ldd bin/xmlback

-- ud

rodig03
Posts: 44
Joined: Thu Sep 18, 2014 1:27 pm

Re: python3 defunct processes with version 20.04

Post by rodig03 » Wed Aug 26, 2020 3:12 pm

Hi,
just updating to 20.04.000.185 did not change anything.
Seems some libraries are missing.

Code: Select all

[openemm@nl ~]$ ldd bin/xmlback
        linux-vdso.so.1 =>  (0x00007ffeeef48000)
        libsqlite3.so.0 => /lib64/libsqlite3.so.0 (0x00007f8d8467c000)
        libssl.so.1.1 => not found
        libcrypto.so.1.1 => not found
        libxml2.so.2 => /lib64/libxml2.so.2 (0x00007f8d84312000)
        libz.so.1 => /lib64/libz.so.1 (0x00007f8d840fc000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f8d83ef8000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f8d83bf6000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f8d83828000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f8d8360c000)
        liblzma.so.5 => /lib64/liblzma.so.5 (0x00007f8d833e6000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f8d84931000)
[openemm@nl ~]$

ud
Posts: 153
Joined: Thu Aug 17, 2006 9:56 am
Location: Munich, Germany
Contact:

Re: python3 defunct processes with version 20.04

Post by ud » Thu Aug 27, 2020 12:52 pm

Hello,

which operation system in which version are you using, please?

-- ud

rodig03
Posts: 44
Joined: Thu Sep 18, 2014 1:27 pm

Re: python3 defunct processes with version 20.04

Post by rodig03 » Thu Aug 27, 2020 4:04 pm

Hello,
this is centOS 7.
[root@nl ~]# cat /etc/centos-release
CentOS Linux release 7.8.2003 (Core)

[root@nl ~]# uname -a
Linux xxx 3.10.0-1127.18.2.el7.x86_64 #1 SMP Sun Jul 26 15:27:06 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

rodig03
Posts: 44
Joined: Thu Sep 18, 2014 1:27 pm

Re: python3 defunct processes with version 20.04

Post by rodig03 » Fri Aug 28, 2020 6:13 pm

Hi,
do I need to upgrade to CentOS 8 to get openssl 1.1 and get the system running?
Or is there a simpler way?
Seems you are relying on having version 1.1
Thanks

Robert

Post Reply