New Install openemm-runtime-20.10.070 - Python3 Error

[RayS2020]

Centos8.2
Open Source Runtime-20.10.070

Followed Directions in OpenEMM 20.10 Install & Admin Guide then applied Chapter 12 - Deployment of Python 3.8 (or later).
When running OMT.sh with Root or Openemm user account - this error message appears:

[root@cmbropenemm openemm]# /home/openemm/bin/OMT.sh
Checking python installation ...
Python version found: 3.8.6
Mandatory python mysql module is missing!
Command to install python mysql module: 'sudo yum -y install MySQL-python'
Install python mysql module now? Y/n n
Mandatory python mysql module is still missing!

Selected Yes the first time around and the install still failed.

Then issued this command and this is the failure I am seeing:
Command:
python3 -m pip install mysqlclient

Error:
Command "/usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-bjmm40ij/mysqlclient/setup.py';f=getattr(tokenize, 'open',
open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-fy9n0c_u-record/install-
record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-bjmm40ij/mysqlclient/


Any thoughts on how to resolve this failure?

Thank you,

/ray

[JuergenB]

You could try this bash script (not perfect, but i works) with CentOS 8.2 minimum features.
It should get you just before the download and install of OpenEMM 20.10

Code: Select all

# Before using / starting this script
# Install a minimal CentOs from CCentOS-8.2.2004-x86_64-boot.iso
# Modify passwords used in this script

# Start
sudo su root

# Variables

OpenEMM="20.10"
PythonRelease="3.9.0"
OpenEmmPass="Password123"
MariaDBPass="Password123"

# Verifiy if cron is running
# su - root -c "service crond status;exit"

# Verifiy is SELinux is Disabled
getenforce

# Permissive : Actions contrary to the policy are only logged in the audit log.
su - root -c "sed -i 's/SELINUX=permissive/SELINUX=disabled/g' /etc/selinux/config"
su - root -c "sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config"


# 3.2 Packages of Operating System
dnf -y install net-tools wget nano zip tar
dnf -y update
dnf -y install gcc make 
dnf -y install java-11-openjdk-devel
dnf -y install xorg-x11-fonts-75dpi zlib fontconfig 
dnf -y install freetype libX11 libXext libXrender urw-fonts

# 3.4 DBMS / MariaDB

# <!-- Compile Python 3.9.0
# OpenEMM 20.10, Pyhton 3.9.0 required, lets compile (Admin Guide 1.3.2, Page 12).
# If you use OpenEMM 20.10 you need Python 3.8 (or later). See chapter 13 for details.
# Check here -> https://www.python.org/ftp/python/ and set Release Version in $PythonRelease

dnf -y install wget gcc gcc-c++ bzip2-devel
dnf -y install gdbm-devel libgcrypt-devel libffi-devel libxml2-devel ncurses-devel
dnf -y install openssl-devel readline-devel sqlite-devel zlib-devel xz xz-devel
dnf -y install expat-devel tk-devel # maybe not used ...  
dnf -y install openssl openssl-devel apr-devel
dnf -y install redhat-rpm-config

# create openemm user
 
groupadd openemm
useradd -m -g openemm -d /home/openemm -s /bin/bash openemm
echo $OpenEmmPass | passwd --stdin openemm
chmod 711 /home/openemm

cd ~
mkdir -p /root/python3
wget -P /root/python3/ https://www.python.org/ftp/python/"$PythonRelease"/Python-"$PythonRelease".tgz

su - root -c "mkdir -p /home/openemm/opt/Python-"$PythonRelease";exit"

chmod 775 /home/openemm/opt
chown openemm:openemm /home/openemm/opt
chown openemm:openemm /home/openemm/opt/* --recursive

su - root -c "rm -f /home/openemm/opt/python3"
su - root -c "ln -s /home/openemm/opt/Python-"$PythonRelease" /home/openemm/opt/python3"
su - root -c "tar -C /root/python3 -xaf /root/python3/Python-"$PythonRelease".tgz"

cd /root/python3/Python-"$PythonRelease"
./configure --prefix=/home/openemm/opt/Python-"$PythonRelease" --enable-optimizations
make -j $(nproc)
# make test  
make install

# cat /root/.bash_profile
# export PATH="/home/openemm/opt/python3/bin:$PATH"
sed -i '/PATH=$PATH/aPATH=$PATH:/home/openemm/opt/python3/bin' /root/.bash_profile
source ~/.bash_profile
echo $PATH
which python3

python3 -m pip install --upgrade pip
python3 -m pip install wheel
python3 -m pip install pyOpenSSL
python3 -m pip install py3dns
python3 -m pip install xlrd xlwt xlutils
python3 -m pip install paramiko pyspf dnspython dkimpy
python3 -m pip install pycrypto
python3 -m pip install requests
python3 -m pip install httpie
python3 -m pip install setproctitle
python3 -m pip install inotify
python3 -m pip install aiodns aiohttp aiohttp-xmlrpc aiosmtpd

# Compile Python -->

# If you use RedHat 8, install MariaDB with 
dnf -y install mariadb-server

# If you use OpenEMM 20.04, install the required MariaDB API and the python driver with
dnf -y install mariadb-devel
python3 -m pip install mysqlclient

# RedHat 8: To start MariaDB and to make sure it starts every time the server is rebooted:
sudo systemctl enable mariadb
sudo systemctl start mariadb

3.8 MySQL/MariaDB Configuration

sudo mysqladmin -u root password $MariaDBPass
echo $MariaDBPass > /root/.mysqlpw  
chmod 600 /root/.mysqlpw 

# Here we will add our content one line after the string is matched
sed -i '/socket=/a\lower_case_table_names=1\nwait_timeout=604800\nsql-mode="STRICT_ALL_TABLES,NO_ENGINE_SUBSTITUTION"' /etc/my.cnf.d/mariadb-server.cnf
 
sudo systemctl restart mariadb

# 4 Server Preparations

# 4.1 Firewall

firewall-cmd --get-active-zones
firewall-cmd --zone=public --add-port=8080/tcp --permanent
firewall-cmd --zone=public --add-port=25/tcp --permanent
firewall-cmd --zone=public --add-forward-port=port=80:proto=tcp:toport=8080 --permanent
firewall-cmd --zone=public --add-forward-port=port=443:proto=tcp:toport=8443 --permanent
firewall-cmd --reload

# 4.2 Postfix Deployment
# just a simple postfix, needs tweaking later

systemctl stop sendmail
dnf -y remove sendmail
dnf -y install postfix sendmail-milter procmail
alternatives --set mta /usr/sbin/sendmail.postfix

# 4.3 Sendmail Deployment
# not used

# 4.4 Tomcat Deployment

# 4.5 Image and PDF Creation Tool wkhtmltox

FILE="/tmp/wkhtmltox-0.12.6-1.centos8.x86_64.rpm"

if [ -f $FILE ];then
    echo "$FILE exists"
    su - root -c "cd /tmp ; rpm -ihv wkhtmltox-*.rpm; exit"	
else
    echo "$FILE doesn't exist"
    su - root -c "cd /tmp ; wget https://github.com/wkhtmltopdf/packaging/releases/download/0.12.6-1/wkhtmltox-0.12.6-1.centos8.x86_64.rpm; rpm -ihv wkhtmltox-*.rpm; exit"
fi

# 4.6 Configuration of Operating System Logging Parameters ?!?
su - root -c "sed  -i 's/       StateFile=\"imjournal.state\") \# File to store the position in the journal/       StateFile=\"imjournal.state\" ratelimit.interval=\"10\" ratelimit.burst=\"10000\")/g' /etc/rsyslog.conf"

su - root -c "systemctl restart rsyslog" 

# 4.7 Miscellaneous
# System value for maximum parallel files open (= ulimit) is 1024. Must be at least 16384.
su - root -c "sed -i '/# End of file/a* hard nofile 16384\n* soft nofile 16384\nroot hard nofile 16384\nroot soft nofile 16384' /etc/security/limits.conf"
su - root -c "sed -i '/# End of file/iopenemm          hard    nofile          16384\nopenemm          soft    nofile          16384\n' /etc/security/limits.conf"
su - root -c "echo 'DefaultLimitNOFILE=16384' >> /etc/systemd/user.conf"
su - root -c "echo 'DefaultLimitNOFILE=16384' >>/etc/systemd/system.conf"

[RayS2020]

Thank you for the script.
I will tweak the script and try it out today.
Will report back with positive feedback shortly.

/ray

[maschoff]

The error you reported has nothing to do with OpenEMM, but with Python 3, pip and the mysql database driver. You could google the error message for help. My hunch is that some lib might be missing.

[RayS2020]

Reinstalled the prerequisites but the issue still exist.
Searched Google but many of the solutions call for updates to PIP.
The correct version of PIP and Python 3.9.0 s in place in place..
Anyone successfully followed the installed instructions and have OpenEMM20.10.070 with CentOS 8.2 running?

While I understand this is not an OpenEMM error - there are no specific fixes online to addresses this missing module error displayed when I run /home/openemm/bin/OMT.sh

[root@cmbropenemmsql bin]# /home/openemm/bin/OMT.sh
Checking python installation ...
Python version found: 3.9.0
Mandatory python mysql module is missing!
Command to install python mysql module: 'sudo yum -y install MySQL-python'
Install python mysql module now? Y/n

FYI - this weekend I performed a clean install of CentOS 8, followed the OpenEMM install instructions, Installed and created Python 3.9.0 per Chapter 12 and continue to see the original error initially reported. Also, tried with Python 3.8.6

Thanks again for your help on this issue.

/ray

[maschoff]

Code: Select all

Command to install python mysql module: 'sudo yum -y install MySQL-python'
Install python mysql module now? Y/n 

Just to make sure: Did you execute "sudo yum -y install MySQL-python" afterwards on your server?

[RayS2020]

Yes - This was performed as well but failed to install. Will post error message shortly..

-Ray

[RayS2020]

Here are the results I am seeing:

[root@cmbropenemmsql ~]# /home/openemm/bin/OMT.sh
Checking python installation ...
Python version found: 3.9.0
Mandatory python mysql module is missing!
Command to install python mysql module: 'sudo yum -y install MySQL-python'
Install python mysql module now? Y/n Y
Last metadata expiration check: 3:01:01 ago on Mon 07 Dec 2020 04:14:39 AM EST.
No match for argument: MySQL-python
Error: Unable to find a match: MySQL-python
Mandatory python mysql module is still missing!
[root@cmbropenemmsql ~]#



Also - If I issue the command "sudo yum -y install MySQL-python" - I get this error message:

[root@cmbropenemmsql ~]# sudo yum -y install MySQL-python
Last metadata expiration check: 0:29:51 ago on Mon 07 Dec 2020 07:17:29 AM EST.
No match for argument: MySQL-python
Error: Unable to find a match: MySQL-python
[root@cmbropenemmsql ~]#

[maschoff]

Unfortunately, I can not see what happens when you enter

Code: Select all

sudo yum -y install MySQL-python

[RayS2020]

here is the error:

No match for argument: MySQL-python
Error: Unable to find a match: MySQL-python

[maschoff]

You could try package python-PyMySQL instead.

[RayS2020]

Tried the command "sudo yum -y install python-PyMySQL" and "sudo yum -y install python3-PyMySQL".

The first command attempted to download the package [see details #1 below] but returned an error saying it cannot download the package.

The second command installed the module [See details #2 below] but the command /home/openemm/bin/OMT.sh still fails with the error "Mandatory python mysql module is missing!".


+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
#1 Detailed Response:

[root@cmbropenemmsql ~]# sudo yum -y install python3-PyMySQL
Last metadata expiration check: 1:26:46 ago on Mon 07 Dec 2020 07:17:29 AM EST.
Dependencies resolved.
================================================================================
Package Arch Version Repo Size
================================================================================
Installing:
python3-PyMySQL noarch 0.8.0-10.module_el8.1.0+245+c39af44f AppStream 159 k
Enabling module streams:
python36 3.6

Transaction Summary
================================================================================
Install 1 Package

Total download size: 159 k
Installed size: 661 k
Downloading Packages:
[MIRROR] python3-PyMySQL-0.8.0-10.module_el8.1.0+245+c39af44f.noarch.rpm: Status code: 404 for http://mirror.steadfastnet.com/centos/8 ... noarch.rpm (IP: 208.100.4.53)
[MIRROR] python3-PyMySQL-0.8.0-10.module_el8.1.0+245+c39af44f.noarch.rpm: Status code: 404 for http://mirrors.oit.uci.edu/centos/8.3.2 ... noarch.rpm (IP: 128.200.84.11)
[MIRROR] python3-PyMySQL-0.8.0-10.module_el8.1.0+245+c39af44f.noarch.rpm: Status code: 404 for http://bay.uchicago.edu/centos/8.3.2011 ... noarch.rpm (IP: 128.135.20.9)
[MIRROR] python3-PyMySQL-0.8.0-10.module_el8.1.0+245+c39af44f.noarch.rpm: Status code: 404 for http://mirrors.liquidweb.com/CentOS/8.3 ... noarch.rpm (IP: 69.167.187.144)
[MIRROR] python3-PyMySQL-0.8.0-10.module_el8.1.0+245+c39af44f.noarch.rpm: Status code: 404 for http://repo1.sea.innoscale.net/centos/8 ... noarch.rpm (IP: 71.19.224.254)
[MIRROR] python3-PyMySQL-0.8.0-10.module_el8.1.0+245+c39af44f.noarch.rpm: Status code: 404 for http://mirrors.sonic.net/centos/8.3.201 ... noarch.rpm (IP: 157.131.0.16)
[MIRROR] python3-PyMySQL-0.8.0-10.module_el8.1.0+245+c39af44f.noarch.rpm: Status code: 404 for http://mirror.twinlakes.net/centos/8.3. ... noarch.rpm (IP: 63.135.176.20)
[MIRROR] python3-PyMySQL-0.8.0-10.module_el8.1.0+245+c39af44f.noarch.rpm: Status code: 404 for http://mirror.cs.uwp.edu/pub/centos/8.3 ... noarch.rpm (IP: 131.210.12.35)
[MIRROR] python3-PyMySQL-0.8.0-10.module_el8.1.0+245+c39af44f.noarch.rpm: Status code: 404 for http://ftpmirror.your.org/pub/centos/8. ... noarch.rpm (IP: 204.9.55.82)
[MIRROR] python3-PyMySQL-0.8.0-10.module_el8.1.0+245+c39af44f.noarch.rpm: Status code: 404 for http://mirror.trouble-free.net/centos/8 ... noarch.rpm (IP: 66.45.230.130)
[FAILED] python3-PyMySQL-0.8.0-10.module_el8.1.0+245+c39af44f.noarch.rpm: No more mirrors to try - All mirrors were already tried without success

The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'yum clean packages'.
Error: Error downloading packages:
Cannot download Packages/python3-PyMySQL-0.8.0-10.module_el8.1.0+245+c39af44f.noarch.rpm: All mirrors were tried

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++


#2 Detailed Response:

Results for this command: sudo yum -y install python3-PyMySQL

[root@cmbropenemmsql ~]# sudo yum -y install python3-PyMySQL
Last metadata expiration check: 0:15:16 ago on Mon 07 Dec 2020 08:44:57 AM EST.
Dependencies resolved.
=================================================================================================
Package Arch Version Repository Size
=================================================================================================
Installing:
python3-PyMySQL noarch 0.8.0-10.module_el8.3.0+562+e162826a AppStream 159 k
Enabling module streams:
python36 3.6

Transaction Summary
=================================================================================================
Install 1 Package

Total download size: 159 k
Installed size: 661 k
Downloading Packages:
python3-PyMySQL-0.8.0-10.module_el8.3.0+562+e162826a.noarch.rpm 65 kB/s | 159 kB 00:02
-------------------------------------------------------------------------------------------------
Total 59 kB/s | 159 kB 00:02
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Installing : python3-PyMySQL-0.8.0-10.module_el8.3.0+562+e162826a.noarch 1/1
Running scriptlet: python3-PyMySQL-0.8.0-10.module_el8.3.0+562+e162826a.noarch 1/1
Verifying : python3-PyMySQL-0.8.0-10.module_el8.3.0+562+e162826a.noarch 1/1

Installed:
python3-PyMySQL-0.8.0-10.module_el8.3.0+562+e162826a.noarch

Complete!

[JuergenB]

Hi,

can you try to install with dnf ?

It is the default package manager of Fedora 22, CentOS8 and RHEL8. DNF is the next generation version of YUM and intended to be the replacement for YUM in RPM-based systems.

I have no problems with my script and CentOs 8 so far.

[RayS2020]

Thank you for your response.

Just so I am 100% clear on this - are you saying you successfully installed OpenEMM on Centos8 using DNF rather than YUM - resulting in no Python error messages?

You also followed the install instructions provided with the OpenEMM 20.10.070 - correct?


Thanks again,

/ray

[JuergenB]

Hi ray,

i have a running CentOs 8 with Python 3.9.0, Java 11, MariaDB and OpenEMM 20.10.*

I started with the old 19.x Release and migrated / upgraded several times. Most of the time i did a fresh install on a clean evironment.

I must admit, that i worked through the Admin Guide 1.3.2 for 20.04 and it´s very strange.
The author tries to get all possible versions (centos 7, java old and new, differen OS or SQL servers) into a single pdf.

But that´s ok, as long as you follow the guide.
I am not really shure if the author has tested this with CentOS. I have used the OVA files for a while and get strange errors in some situation.
Currently a clean CentOS work better for me.

I had some problems during the first setups, because in my opion there is a slightly better way to get this compiled.
I didn´t verified 20.10 install pdf.

For expample i created some user directory (mainly /home/openemm/opt/Python-3.9.0 because of some problems during compilation,
can´t remember.

But IT this works, with one exception (i will do a clean setup today for migration)....

• compile pyhton 3.9.0

• install mariadb

• install MariaDB API and the python driver

I compile pyhton before DBMS (that´s what is workfing for me) AND

at the moment i use a single script, that do all the basic setup (takes about 15-20 minutes on my old i7).
I only need to run the OpenEMM installer and set all the parameters and i am good to go.

oh, smtp is handled extra by me...