linspector/uplink
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

First real UNIX daemon based version of the code. 1000 Cleanups and Fixes... :)

Browse source
This commit is contained in:
Johannes Findeisen 2021-11-13 07:30:05 +01:00
commit 3e5222f97c
7 changed files with 380 additions and 132 deletions
Download patch file Download diff file Expand all files Collapse all files

4
.gitignore vendored
View file

@ -1,4 +1,6 @@
.idea
.run
__pycache__
config.json
uplink.iml
log/*.log*
uplink.iml

18
README.md
View file

@ -1,6 +1,6 @@
# uplink - is not some program expecting uplinks to work!
**This README file is not up to date! It will be fixed soon to explain the installation and the usage of uplink.**
**This README file is not up-to-date! It will be fixed soon to explain the installation and the usage of uplink.**
uplink is a tool to monitor the uplink status of AVM FRITZ!Box Cable and DSL based routers. It uses the TR-064 protocol over UPnP.
@ -12,7 +12,7 @@ For now I can say that uplink can monitor the status of you FRITZ!Box uplinks. I
- Git (only for installing from Git repository)
- Python3 (running for me using 3.7.3 on Raspbian 10 and 3.8.5 on Arch Linux)
- fritzconnection >= 1.3.4
- fritzconnection >= 1.5.0
- pymysql >= 0.10.1
Just install the 3rd party dependencies using `pip install $PACKAGE` or your OS package manager
@ -69,17 +69,3 @@ Edit config.json to your needs.
I actually use "[DBeaver](https://dbeaver.io/)" for taking a look at the data uplink is collecting.
There will be a Gtk+ frontend to uplink at some time but this project is at a very early stage of development, so I want to write the collector first. Even support for other SQL databases is in planning in conjunction with the Gtk+ frontend. I use MariaDB only because I can move fast-forward.
## TODO (in no particular order)
- A lot... :)
- Make all config vars as ARGS and vice versa. ARGS have higher priority. Chain: default -> config -> ARGS.
- ~~Parallelize queries using threads to improve performance; Does not work using SQLite because of exclusive access to the database~~ **DONE**
- ~~SQL server backend; MariaDB/MySQL?~~ **DONE**
- Bring back SQLite support
- Switch to ORM (peewee?) to support PostgreSQL, MySQL and SQLite.
- Gtk+ frontend connecting to the database or loading a local copy of a SQLite file. wxGlade?
- ~~A cron mode to not let uplink run in an endless loop to be scheduled and executed by cron.~~ **DONE**
- A daemon mode to be a real UNIX daemon. For now, it's just sleep() based.
- Always keep platform independence in mind but not if uplink looses nice features on Linux.

112
daemon.py Normal file
View file

@ -0,0 +1,112 @@
"""
Original Source:
https://web.archive.org/web/20131017130434/http://www.jejik.com/articles/2007/02/a_simple_unix_linux_daemon_in_python/
https://web.archive.org/web/20131017130434/http://www.jejik.com/files/examples/daemon3x.py
See daemon3x.py in the uplink source tree.
More Information:
https://franklingu.github.io/programming/2016/03/01/creating-daemon-process-python-example-explanation/
https://dpbl.wordpress.com/2017/02/12/a-tutorial-on-python-daemon/
"""
import sys
import os
import time
import atexit
import signal
# TODO.md: Check why pid files are not being deleted when sending the TERM signal
class Daemon:
def __init__(self, pid_file):
self.pid_file = pid_file
def daemonize(self):
try:
pid = os.fork()
if pid > 0:
sys.exit(0)
except OSError as err:
sys.stderr.write('fork #1 failed: {0}\n'.format(err))
sys.exit(1)
os.chdir('/')
os.setsid()
os.umask(0)
try:
pid = os.fork()
if pid > 0:
sys.exit(0)
except OSError as err:
sys.stderr.write('fork #2 failed: {0}\n'.format(err))
sys.exit(1)
sys.stdout.flush()
sys.stderr.flush()
si = open(os.devnull, 'r')
so = open(os.devnull, 'a+')
se = open(os.devnull, 'a+')
os.dup2(si.fileno(), sys.stdin.fileno())
os.dup2(so.fileno(), sys.stdout.fileno())
os.dup2(se.fileno(), sys.stderr.fileno())
atexit.register(self.del_pid)
pid = str(os.getpid())
with open(self.pid_file, 'w+') as f:
f.write(pid + '\n')
def del_pid(self):
os.remove(self.pid_file)
def start(self):
try:
with open(self.pid_file, 'r') as pf:
pid = int(pf.read().strip())
except IOError:
pid = None
if pid:
message = "pid_file {0} already exist. Daemon already running?\n"
sys.stderr.write(message.format(self.pid_file))
sys.exit(1)
self.daemonize()
self.run()
def stop(self):
try:
with open(self.pid_file, 'r') as pf:
pid = int(pf.read().strip())
except IOError:
pid = None
if not pid:
message = "pid_file {0} does not exist. Daemon not running?\n"
sys.stderr.write(message.format(self.pid_file))
return
try:
while 1:
os.kill(pid, signal.SIGTERM)
time.sleep(0.1)
except OSError as err:
e = str(err.args)
if e.find("No such process") > 0:
if os.path.exists(self.pid_file):
os.remove(self.pid_file)
else:
print(str(err.args))
sys.exit(1)
def restart(self):
self.stop()
self.start()
def run(self):
"""Override this method when you subclass Daemon.
It will be called after the process has been daemonized by
start() or restart()."""

122
daemon3x.py Normal file
View file

@ -0,0 +1,122 @@
"""Generic linux daemon base class for python 3.x."""
import sys, os, time, atexit, signal
class daemon:
"""A generic daemon class.
Usage: subclass the daemon class and override the run() method."""
def __init__(self, pidfile): self.pidfile = pidfile
def daemonize(self):
"""Deamonize class. UNIX double fork mechanism."""
try:
pid = os.fork()
if pid > 0:
# exit first parent
sys.exit(0)
except OSError as err:
sys.stderr.write('fork #1 failed: {0}\n'.format(err))
sys.exit(1)
# decouple from parent environment
os.chdir('/')
os.setsid()
os.umask(0)
# do second fork
try:
pid = os.fork()
if pid > 0:
# exit from second parent
sys.exit(0)
except OSError as err:
sys.stderr.write('fork #2 failed: {0}\n'.format(err))
sys.exit(1)
# redirect standard file descriptors
sys.stdout.flush()
sys.stderr.flush()
si = open(os.devnull, 'r')
so = open(os.devnull, 'a+')
se = open(os.devnull, 'a+')
os.dup2(si.fileno(), sys.stdin.fileno())
os.dup2(so.fileno(), sys.stdout.fileno())
os.dup2(se.fileno(), sys.stderr.fileno())
# write pidfile
atexit.register(self.delpid)
pid = str(os.getpid())
with open(self.pidfile,'w+') as f:
f.write(pid + '\n')
def delpid(self):
os.remove(self.pidfile)
def start(self):
"""Start the daemon."""
# Check for a pidfile to see if the daemon already runs
try:
with open(self.pidfile,'r') as pf:
pid = int(pf.read().strip())
except IOError:
pid = None
if pid:
message = "pidfile {0} already exist. " + \
"Daemon already running?\n"
sys.stderr.write(message.format(self.pidfile))
sys.exit(1)
# Start the daemon
self.daemonize()
self.run()
def stop(self):
"""Stop the daemon."""
# Get the pid from the pidfile
try:
with open(self.pidfile,'r') as pf:
pid = int(pf.read().strip())
except IOError:
pid = None
if not pid:
message = "pidfile {0} does not exist. " + \
"Daemon not running?\n"
sys.stderr.write(message.format(self.pidfile))
return # not an error in a restart
# Try killing the daemon process
try:
while 1:
os.kill(pid, signal.SIGTERM)
time.sleep(0.1)
except OSError as err:
e = str(err.args)
if e.find("No such process") > 0:
if os.path.exists(self.pidfile):
os.remove(self.pidfile)
else:
print (str(err.args))
sys.exit(1)
def restart(self):
"""Restart the daemon."""
self.stop()
self.start()
def run(self):
"""You should override this method when you subclass Daemon.
It will be called after the process has been daemonized by
start() or restart()."""

26
test.py Normal file
View file

@ -0,0 +1,26 @@
#!/usr/bin/python3 -d
"""
Daemon testing file... Daemons in Python are new to me because I only worked on CLI, Gtk+ and Django projects before.
Let's have some fun... :)
"""
from daemon import Daemon
class Test(Daemon):
def __init__(self, pid_file, color, size):
# super().__init__(pid_file)
self.pid_file = pid_file
self.color = color
self.size = size
def run(self):
# Write my daemon test code here:
return
Test = Test("/tmp/uplink.pid", "Red", "XXL")
Test.start()

136
uplink
View file

@ -21,32 +21,23 @@
# OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
import argparse
import calendar
import json
import logging
import logging.handlers
import os
import os.path as path
import pymysql
import socket
import sys
import time
from fritzconnection.lib.fritzstatus import FritzStatus
from threading import Thread
from uplink import Uplink
# Version format: MAJOR.FEATURE.FIXES
__version__ = "0.2.3"
__version__ = "0.3.0-development"
logger = logging.getLogger(__name__)
PURPLE = "\033[95m"
BLUE = "\033[94m"
YELLOW = "\033[93m"
GRAY = "\033[90m"
END = "\033[0m"
# TODO.md: Implement logging
# TODO.md: Implement --no-daemon for a single run e.g. "cron mode"
# TODO.md: Implement CLI output messages
# TODO.md: IDEA: Implement a small webserver inline to get statistics and graphs over the network? Or maybe better as a
# separate daemon
# TODO.md: Make use of the args passed to the script
# TODO.md: Cleanup args and reduce to only what makes sense
def parse_args():
parser = argparse.ArgumentParser(
description="uplink is a tool to monitor the link status of AVM FRITZ!Box Cable and DSL based routers.",
@ -76,25 +67,25 @@ def parse_args():
parser.add_argument("-l", "--logfile", metavar="FILE", help="logfile to use")
parser.add_argument("-c", "--logcount", default=5, type=int,
parser.add_argument("-c", "--log-count", default=5, type=int,
help="maximum number of logfiles in rotation (default: 5)")
parser.add_argument("-m", "--logsize", default=10485760, type=int,
parser.add_argument("-m", "--log-size", default=10485760, type=int,
help="maximum logfile size in bytes (default: 10485760)")
parser.add_argument("-s", "--stdout", default=False, dest="stdout", action="store_true", help="log to stdout")
output = parser.add_mutually_exclusive_group()
output.add_argument("-q", "--quiet", action="store_const", dest="loglevel", const=logging.ERROR,
output.add_argument("-q", "--quiet", action="store_const", dest="log_level", const=logging.ERROR,
help="output only errors")
output.add_argument("-w", "--warning", action="store_const", dest="loglevel", const=logging.WARNING,
output.add_argument("-w", "--warning", action="store_const", dest="log_level", const=logging.WARNING,
help="output warnings")
output.add_argument("-v", "--verbose", action="store_const", dest="loglevel", const=logging.INFO,
output.add_argument("-v", "--verbose", action="store_const", dest="log_level", const=logging.INFO,
help="output info messages")
output.add_argument("-d", "--debug", action="store_const", dest="loglevel", const=logging.DEBUG,
output.add_argument("-d", "--debug", action="store_const", dest="log_level", const=logging.DEBUG,
help="output debug messages")
output.set_defaults(loglevel=logging.ERROR)
@ -103,106 +94,21 @@ def parse_args():
return parser.parse_args()
def get_data(config, inc):
try:
con = pymysql.connect(host=config["database_host"],
user=config["database_user"],
password=config["database_password"],
database=config["database"])
except Exception as err:
logger.error("Database connection failed: " + str(err))
exit(1)
timestamp = calendar.timegm(time.gmtime())
local_time = time.localtime(timestamp)
d = time.strftime("%Y-%m-%d ", local_time)
t = time.strftime("%H:%M:%S", local_time)
try:
fc = FritzStatus(address=config["uplinks"][inc]["ip"], password=config["uplinks"][inc]["password"])
except Exception as err:
logger.error(str(err) + " on device with ip: " + config["uplinks"][inc]["ip"])
sql = 'INSERT INTO log (timestamp, date, time, internal_ip, is_linked, is_connected, provider, message, ' \
'source_host) VALUES (\"' + str(timestamp) + '\",\"' + str(d) + '\",\"' + str(t) + '\",\"' + \
str(config["uplinks"][inc]["ip"]) + '\",\"' + "0" + '\",\"' + "0" + '\",\"' + \
str(config["uplinks"][inc]["provider"]) + '\",\"ERROR: ' + str(err) + '\",\"' + socket.gethostname() + \
'\")'
logger.debug(sql)
with con.cursor() as cur:
cur.execute(sql)
con.commit()
exit(1)
if fc.is_connected:
status = "UP"
else:
status = "DOWN"
logger.info(config["uplinks"][inc]["provider"] + " " + status)
sql = 'INSERT INTO log (timestamp, date, time, uptime, internal_ip, external_ip, external_ipv6, is_linked, ' \
'is_connected, str_transmission_rate_up, str_transmission_rate_down, str_max_bit_rate_up, ' \
'str_max_bit_rate_down, str_max_linked_bit_rate_up, str_max_linked_bit_rate_down, modelname, ' \
'system_version, provider, message, source_host) VALUES (\"' + str(timestamp) + '\",\"' + str(d) + '\",\"' + \
str(t) + '\",\"' + str(fc.uptime) + '\",\"' + str(config["uplinks"][inc]["ip"]) + '\",\"' + \
str(fc.external_ip) + '\",\"' + str(fc.external_ipv6) + '\",\"' + str(int(fc.is_linked)) + '\",\"' + \
str(int(fc.is_connected)) + '\",\"' + str(fc.str_transmission_rate[0]) + '\",\"' + \
str(fc.str_transmission_rate[1]) + '\",\"' + str(fc.str_max_bit_rate[0]) + '\",\"' + \
str(fc.str_max_bit_rate[1]) + '\",\"' + str(fc.str_max_linked_bit_rate[0]) + '\",\"' + \
str(fc.str_max_linked_bit_rate[1]) + '\",\"' + str(fc.modelname) + '\",\"' + \
str(fc.fc.system_version) + '\",\"' + str(config["uplinks"][inc]["provider"]) + '\",\"' + status + '\",\"' + \
socket.gethostname() + '\")'
logger.debug(sql)
with con.cursor() as cur:
cur.execute(sql)
con.commit()
con.close()
def run(config):
while True:
for i in range(len(config["uplinks"])):
t = Thread(target=get_data, args=(config, i))
t.start()
if config["cron"]:
break
time.sleep(config["interval"])
if __name__ == "__main__":
args = parse_args()
root_logger = logging.getLogger()
if args.stdout:
formatter = logging.Formatter(
PURPLE + "%(asctime)s" + END + ":" + BLUE + "%(levelname)s" + END + ":" + YELLOW + "%(name)s" + END + ":" + GRAY + "%(message)s" + END)
handler1 = logging.StreamHandler(sys.stdout)
handler1.setFormatter(formatter)
root_logger.addHandler(handler1)
if args.logfile:
logfile = path.expanduser(args.logfile)
if not path.exists(path.dirname(logfile)):
os.makedirs(path.dirname(logfile))
formatter = logging.Formatter("%(asctime)s:%(levelname)s:%(name)s:%(message)s")
handler2 = logging.handlers.RotatingFileHandler(args.logfile, maxBytes=args.logsize, backupCount=args.logcount)
handler2.setFormatter(formatter)
root_logger.addHandler(handler2)
root_logger.setLevel(args.loglevel)
# TODO.md: Read --version argument and print program version; then exit.
# TODO.md: Cleanup config.json and only define what really is needed
config_path = args.config
try:
with open(config_path, 'r') as configfile:
config_data = configfile.read()
_config = json.loads(config_data)
except Exception as err:
logger.error("Reading configuration file '" + config_path + "' failed: " + str(err))
# TODO.md: Replace all lines like this with generic Python logging
print(str("Uplink: Configuration Error!"))
exit(1)
try:
run(_config)
except KeyboardInterrupt:
logger.info("Program terminated!")
exit(0)
uplink = Uplink("/tmp/uplink.pid", _config)
uplink.start()

94
uplink.py Normal file
View file

@ -0,0 +1,94 @@
import pymysql
import socket
import time
import calendar
from fritzconnection.lib.fritzstatus import FritzStatus
from threading import Thread
from daemon import Daemon
class Uplink(Daemon):
def __init__(self, pid_file, config):
self.pid_file = pid_file
self.config = config
@staticmethod
def get_data(config, inc):
try:
# TODO.md: Think about using a DB ORM (SQLAlchemy?) to make this program supporting different databases like
# sqlite and Postgres
con = pymysql.connect(host=config["database_host"],
user=config["database_user"],
password=config["database_password"],
database=config["database"])
except Exception as err:
# TODO.md: Replace all lines like this with generic Python logging
print(str("Uplink: Database connection failed: " + str(err)))
exit(1)
timestamp = calendar.timegm(time.gmtime())
local_time = time.localtime(timestamp)
d = time.strftime("%Y-%m-%d ", local_time)
t = time.strftime("%H:%M:%S", local_time)
try:
fc = FritzStatus(address=config["uplinks"][inc]["ip"], password=config["uplinks"][inc]["password"])
except Exception as err:
# TODO.md: Replace all lines like this with generic Python logging
print(str(str(err) + " on device with ip: " + config["uplinks"][inc]["ip"]))
# TODO.md: Fix to long lines and make the SQL statement more readable
sql = 'INSERT INTO log (timestamp, date, time, internal_ip, is_linked, is_connected, provider, message, ' \
'source_host) VALUES (\"' + str(timestamp) + '\",\"' + str(d) + '\",\"' + str(t) + '\",\"' + \
str(config["uplinks"][inc]["ip"]) + '\",\"' + "0" + '\",\"' + "0" + '\",\"' + \
str(config["uplinks"][inc]["provider"]) + '\",\"ERROR: ' + str(err) + '\",\"' + socket.gethostname() + \
'\")'
# TODO.md: Replace all lines like this with generic Python logging
print(str(sql))
with con.cursor() as cur:
cur.execute(sql)
con.commit()
exit(1)
if fc.is_connected:
status = "UP"
else:
status = "DOWN"
# TODO.md: Replace all lines like this with generic Python logging
print(str(config["uplinks"][inc]["provider"] + " " + status))
# TODO.md: Fix to long lines and make the SQL statement more readable
sql = 'INSERT INTO log (timestamp, date, time, uptime, internal_ip, external_ip, external_ipv6, is_linked, ' \
'is_connected, str_transmission_rate_up, str_transmission_rate_down, str_max_bit_rate_up, ' \
'str_max_bit_rate_down, str_max_linked_bit_rate_up, str_max_linked_bit_rate_down, modelname, ' \
'system_version, provider, message, source_host) VALUES (\"' + str(timestamp) + '\",\"' + str(d) + '\",\"' + \
str(t) + '\",\"' + str(fc.uptime) + '\",\"' + str(config["uplinks"][inc]["ip"]) + '\",\"' + \
str(fc.external_ip) + '\",\"' + str(fc.external_ipv6) + '\",\"' + str(int(fc.is_linked)) + '\",\"' + \
str(int(fc.is_connected)) + '\",\"' + str(fc.str_transmission_rate[0]) + '\",\"' + \
str(fc.str_transmission_rate[1]) + '\",\"' + str(fc.str_max_bit_rate[0]) + '\",\"' + \
str(fc.str_max_bit_rate[1]) + '\",\"' + str(fc.str_max_linked_bit_rate[0]) + '\",\"' + \
str(fc.str_max_linked_bit_rate[1]) + '\",\"' + str(fc.modelname) + '\",\"' + \
str(fc.fc.system_version) + '\",\"' + str(config["uplinks"][inc]["provider"]) + '\",\"' + status + '\",\"' + \
socket.gethostname() + '\")'
# TODO.md: Replace all lines like this with generic Python logging
print(str(sql))
with con.cursor() as cur:
cur.execute(sql)
con.commit()
con.close()
def get_config(self):
return self.config
def run(self):
while True:
for i in range(len(self.config["uplinks"])):
t = Thread(target=self.get_data, args=(self.get_config(), i))
t.start()
if self.config["cron"]:
break
time.sleep(self.config["interval"])