args cleanup and full configuration in config file possible now. lot of refactoring and tons of cleanups

This commit is contained in:
Johannes Findeisen 2022-08-29 01:33:19 +02:00
commit d7f74f3914
12 changed files with 157 additions and 199 deletions

109
README.md
View file

@ -1,108 +1,3 @@
# uplink - is not some program expecting uplinks to work!
# uplink -is a tool to monitor the uplink status of AVM FRITZ!Box Cable and DSL based routers.
## **This README file is not up-to-date! It will be fixed soon to explain the installation and the usage of uplink.**
uplink is a tool to monitor the uplink status of AVM FRITZ!Box Cable and DSL based routers. It uses the TR-064 protocol over UPnP.
## Features
For now, I can say that uplink can monitor the status of you FRITZ!Box uplinks. It can log to a file and writes results to a MariaDB/MySQL database. With some editing of the code you can you use SQLite too but this feature is not active at the moment. This will change in the future. I switched to MariaDB because I run uplink on a RaspberryPi and are evaluating results on my workstation. I am working on a Gtk+ frontend to generate statistics...
## Requirements
- A Linux based operating system (I don't test any other OS's but Raspbian is fine and a good target)
- Git (only for installing from Git repository)
- Python3 (running for me using 3.7.3, 3.8.5, 3.9.7 and 3.10.1 on Linux)
- fritzconnection >= 1.5.0 (included as submodule in the Git repository)
- pymysql >= 0.10.1
- sqlalchemy = 1.4.27 (not for now)
- urwid = 2.1.2 (not for now)
Just install the 3rd party dependencies using `pip install $PACKAGE` or your OS package manager
## Installation
git clone https://git.unixpeople.org/hanez/uplink.git
cd uplink
# To get a local copy of fritzconnection you need to get the submodules:
git submodule update --init --recursive
cp config.example.json config.json
### Configuration
Edit config.json to your needs.
#### Variables
- interval: The polling interval
- uplinks: List of devices you want to poll
- provider: Custom name of the uplink provider
- ip: The IP of the router device
- password: The password of the router device
#### Example
{
"interval": 60,
"database_type": "mysql", (NOT USED)
"database": "uplink",
"database_host": "127.0.0.1",
"database_user": "USER",
"database_password": "PASSWORD",
"cron": false, (NOT USED)
"daemon": false, (NOT USED)
"uplinks": [
{ "provider": "Cable Provider", "ip": "192.168.0.1", "password": "1234" },
{ "provider": "DSL Provider", "ip": "192.168.1.1", "password": "1234" }
]
}
### Create Database
TODO
For now there is only the file uplink.sql which will create the tables in your MariaDB/MySQL database.
The file is under files/uplink.sql
## Usage
./uplink [-c | -d | -f] ./config.json
### Help
./uplink --help
```
usage: uplink [-h] [-c | -d | -f] [-i INTERVAL] [--version] CONFIGFILE
uplink is a tool to monitor the link status of AVM FRITZ!Box Cable and DSL based routers.
positional arguments:
CONFIGFILE the configfile to use
options:
-h, --help show this help message and exit
-c, --cron cron mode executes the script only once without loop (default: false)
-d, --daemon run as native daemon (default: false)
-f, --foreground run looped in foreground (default: false)
-i INTERVAL, --interval INTERVAL
poll interval in seconds. this overrides config file settings. (default: 60)
--version show program's version number and exit
```
## View collected data
I actually use "[DBeaver](https://dbeaver.io/)" for taking a look at the data uplink is collecting.
There will be a Gtk+ frontend to uplink at some time but this project is at a very early stage of development, so I want to write the collector first. Even support for other SQL databases is in planning in conjunction with the Gtk+ frontend. I use MariaDB only because I can move fast-forward.
## Planned features
- A lot... :)
- Make all config vars as ARGS and vice versa. ARGS have higher priority. Chain: default -> config -> ARGS.
- Bring back SQLite support.
- Switch to ORM (peewee, sqlalchemy?) to support PostgreSQL, MariaDB/MySQL, SQLite and maybe more databases.
- Gtk+/urwid frontend for visualizing the collected data. wxGlade?
- A tool to generate reports for showing to your uplink provider.
- Always keep platform independence in mind but not if uplink looses nice features on Linux.
- Implement a speedtest feature to regularly run speedtest but independent to uptime checks with different interval.
- A small embedded webserver to view what is going on.
Go to [https://hanez.org/uplink.html](https://hanez.org/uplink.html) to view the latest documentation.

View file

@ -71,37 +71,18 @@ def parse_args():
mode.add_argument('-f', '--foreground', default=False, dest='foreground', action='store_true',
help='run looped in foreground (default: false)')
parser.add_argument('--server', default=False, dest='server', action='store_true',
parser.add_argument('--httpserver', default=False, dest='httpserver', action='store_true',
help='start http server for status information and statistics')
parser.add_argument('-i', '--interval', type=int, help='poll interval in seconds. this '
'overrides config file settings. (default: 60)')
'overrides config file settings (default: 60)')
parser.add_argument('-l', '--logfile', metavar='LOGFILE', help='logfile to use')
parser.add_argument('-n', '--logcount', default=5, type=int,
help='maximum number of logfiles in rotation (default: 5)')
parser.add_argument('-m', '--logsize', default=10485760, type=int,
help='maximum logfile size in bytes (default: 1048576)')
parser.add_argument('-l', '--logfile', metavar='LOGFILE', help='logfile to use. this '
'overrides config file settings')
parser.add_argument('-s', '--stdout', default=False, dest='stdout', action='store_true',
help='log to stdout')
output = parser.add_mutually_exclusive_group()
output.add_argument('-q', '--quiet', action='store_const', dest='loglevel',
const=logging.ERROR, help='output only errors')
output.add_argument('-w', '--warning', action='store_const', dest='loglevel',
const=logging.WARNING, help='output warnings')
output.add_argument('-v', '--verbose', action='store_const', dest='loglevel',
const=logging.INFO, help='output info messages')
output.add_argument('-e', '--debug', action='store_const', dest='loglevel',
const=logging.DEBUG, help='output debug messages')
output.set_defaults(loglevel=logging.ERROR)
parser.add_argument('--version', action='version', version='%(prog)s ' + str(__version__))
return parser.parse_args()
@ -112,25 +93,6 @@ def main():
root_logger = logging.getLogger()
if args.logfile:
logfile = os.path.expanduser(args.logfile)
if not os.path.exists(os.path.dirname(logfile)):
os.makedirs(os.path.dirname(logfile))
formatter1 = logging.Formatter('%(asctime)s:%(levelname)s:%(name)s:%(message)s')
handler1 = logging.handlers.RotatingFileHandler(args.logfile, maxBytes=args.logsize,
backupCount=args.logcount)
handler1.setFormatter(formatter1)
root_logger.addHandler(handler1)
if args.stdout:
formatter2 = logging.Formatter('[%(asctime)s] [%(levelname)s] %(message)s')
handler2 = logging.StreamHandler(sys.stdout)
handler2.setFormatter(formatter2)
root_logger.addHandler(handler2)
root_logger.setLevel(args.loglevel)
try:
with open(args.configuration_file, 'r', encoding='utf-8') as configuration_file:
configuration_data = configuration_file.read()
@ -147,9 +109,39 @@ def main():
if args.interval:
configuration.set_interval(args.interval)
configuration.set_env_var('_server', False)
if args.server:
configuration.set_env_var('_server', True)
if args.logfile:
configuration.set_log_file(args.logfile)
if args.logfile or configuration.get_log_file():
log_file = os.path.expanduser(configuration.get_log_file())
if not os.path.exists(os.path.dirname(log_file)):
os.makedirs(os.path.dirname(log_file))
formatter1 = logging.Formatter('%(asctime)s:%(levelname)s:%(name)s:%(message)s')
handler1 = logging.handlers.RotatingFileHandler(log_file,
maxBytes=configuration.get_log_size(),
backupCount=configuration.get_log_count())
handler1.setFormatter(formatter1)
root_logger.addHandler(handler1)
if args.stdout:
formatter2 = logging.Formatter('[%(asctime)s] [%(levelname)s] %(message)s')
handler2 = logging.StreamHandler(sys.stdout)
handler2.setFormatter(formatter2)
root_logger.addHandler(handler2)
# errors will always show up even when no log_level is set!
root_logger.setLevel(logging.ERROR)
log_level = configuration.get_log_level()
if log_level == "warning":
root_logger.setLevel(logging.WARNING)
elif log_level == "verbose":
root_logger.setLevel(logging.INFO)
elif log_level == "debug":
root_logger.setLevel(logging.DEBUG)
if args.httpserver or configuration.get_httpserver():
configuration.set_env_var('_httpserver', True)
try:
u = Uplink(configuration.get_pid_file(), configuration)
@ -171,9 +163,9 @@ def main():
u.start()
elif args.foreground:
from threading import Thread
if configuration.get_env_var('_server'):
from uplink.server_cherrypy import Server
s = Server(configuration)
if configuration.get_env_var('_httpserver'):
from uplink.httpserver import HTTPServer
s = HTTPServer(configuration)
st = Thread(target=s.run_server, daemon=True)
st.start()

View file

@ -1,14 +1,17 @@
{
"interval": 60,
"database_name": "uplink",
"database_host": "127.0.0.1",
"database_user": "USER",
"database_password": "PASSWORD",
"database_name": "uplink",
"database_password": "uplink",
"database_type": "mysql",
"database_user": "uplink",
"httpserver_host": "0.0.0.0",
"httpserver_port": 8080,
"httpserver_framework": "cherrypy",
"mode": "cron",
"httpserver_port": 4000,
"interval": 30,
"log_level": "info",
"log_count": 5,
"log_size": 0,
"pid_file": "/var/run/user/1000/uplink.pid",
"run_mode": "cron",
"uplinks": [
{
"provider": "Cable Provider",
@ -21,4 +24,4 @@
"password": "1234"
}
]
}
}

View file

@ -26,27 +26,27 @@ logger = getLogger(__name__)
class Configuration:
def __init__(self, configuration):
self.__configuration = configuration
self.__database_host = None
self.__database_name = None
self.__database_password = None
self.__database_user = None
self.__httpserver_framework = None
# environment vars which can be set dynamically at runtime.
self.__env = {}
self.__httpserver = False
self.__httpserver_host = '0.0.0.0'
self.__httpserver_port = 8080
self.__interval = '60'
self.__log_count = 5
self.__interval = 60
self.__log_count = 1
self.__log_file = None
self.__log_level = 'info'
self.__log_size = 10485760,
self.__log_level = None
self.__log_size = 0
self.__pid_file = '/tmp/uplink.pid'
self.__run_mode = None
self.__run_mode = 'cron'
self.__uplinks = None
self.__env = {}
if'database_host' in self.__configuration:
if 'database_host' in self.__configuration:
self.__database_host = self.__configuration['database_host']
else:
raise Exception('database_host required!')
@ -66,8 +66,8 @@ class Configuration:
else:
raise Exception('database_user required!')
if 'httpserver_framework' in self.__configuration:
self.__httpserver_framework = self.__configuration['httpserver_framework']
if 'httpserver' in self.__configuration:
self.__httpserver = self.__configuration['httpserver']
if 'httpserver_host' in self.__configuration:
self.__httpserver_host = self.__configuration['httpserver_host']
@ -78,9 +78,21 @@ class Configuration:
if 'interval' in self.__configuration:
self.__interval = self.__configuration['interval']
if 'log_count' in self.__configuration:
self.__log_count = self.__configuration['log_count']
if 'log_file' in self.__configuration:
self.__log_file = self.__configuration['log_file']
if 'log_level' in self.__configuration:
self.__log_level = self.__configuration['log_level']
if 'log_size' in self.__configuration:
self.__log_size = self.__configuration['log_size']
if 'pid_file' in self.__configuration:
self.__pid_file = self.__configuration['pid_file']
if 'run_mode' in self.__configuration:
self.__run_mode = self.__configuration['run_mode']
@ -89,6 +101,7 @@ class Configuration:
else:
raise Exception('uplinks required!')
# get methods
def get_database_host(self):
return self.__database_host
@ -105,13 +118,11 @@ class Configuration:
return self.__env
def get_env_var(self, name):
return self.__env[name]
if name in self.__env:
return self.__env[name]
def set_env_var(self, name, value):
self.__env[name] = value
def get_httpserver_framework(self):
return self.__httpserver_framework
def get_httpserver(self):
return self.__httpserver
def get_httpserver_host(self):
return self.__httpserver_host
@ -122,12 +133,18 @@ class Configuration:
def get_interval(self):
return self.__interval
def set_interval(self, interval):
self.__interval = interval
def get_log_file(self):
return self.__log_file
def get_log_count(self):
return self.__log_count
def get_log_level(self):
return self.__log_level
def get_log_size(self):
return self.__log_size
def get_pid_file(self):
return self.__pid_file
@ -139,3 +156,16 @@ class Configuration:
def get_uplinks(self):
return self.__uplinks
# set methods
def set_env_var(self, name, value):
self.__env[name] = value
def set_httpserver(self, value):
self.__httpserver = value
def set_interval(self, interval):
self.__interval = interval
def set_log_file(self, value):
self.__log_file = value

View file

@ -23,10 +23,10 @@ from logging import getLogger
logger = getLogger(__name__)
class Csv:
class CSV:
def __init__(self, configuration):
self.configuration = configuration
self.__configuration = configuration
def write_model_to_csv(self):
return

View file

@ -29,7 +29,7 @@ from logging import getLogger
logger = getLogger(__name__)
class Server:
class HTTPServer:
def __init__(self, configuration):
self.__configuration = configuration

View file

@ -36,6 +36,7 @@ class Model:
self.__internal_ip = None
self.__is_connected = None
self.__is_linked = None
# TODO: rename message to status
self.__message = None
self.__model_name = None
self.__provider = None

29
uplink/notification.py Normal file
View file

@ -0,0 +1,29 @@
# Copyright (c) 2022 Johannes Findeisen <you@hanez.org>
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is furnished
# to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice (including the next
# paragraph) shall be included in all copies or substantial portions of the
# Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
# FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS
# OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
# WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF
# OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
from logging import getLogger
logger = getLogger(__name__)
class Notification:
def __init__(self, configuration):
self.__configuration = configuration

View file

@ -28,7 +28,8 @@ logger = getLogger(__name__)
class Speedtest:
def __init__(self, configuration):
self.configuration = configuration
self.__configuration = configuration
self.download = None
self.upload = None

View file

@ -18,10 +18,9 @@
# WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF
# OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
# TODO: Switch all SQL stuff to SQLAlchemy using the uplink.database Database class
# TODO: Make the use of a database optional and only output to stdout and/or in a logfile (use rich
# for colorizing the std output (https://github.com/Textualize/rich)); Output to a CSV file should
# be optional too.
# be optional too. Also reinvent SQLite as database backend.
import calendar
import socket
@ -33,6 +32,7 @@ from threading import Thread
from uplink.daemon import Daemon
from uplink.database import Database
from uplink.model import Model
from uplink.notification import Notification
logger = getLogger(__name__)
@ -41,7 +41,8 @@ class Uplink(Daemon):
def __init__(self, pid_file, configuration):
super().__init__(pid_file)
self.configuration = configuration
self.__configuration = configuration
self.date = None
self.time = None
self.status = None
@ -52,19 +53,25 @@ class Uplink(Daemon):
self.date = time.strftime('%Y-%m-%d', local_time)
self.time = time.strftime('%H:%M:%S', local_time)
uplink = self.configuration.get_uplink(i)
self.__configuration.set_env_var('_last_run', self.date + ' ' + self.time)
uplink = self.__configuration.get_uplink(i)
try:
fc = FritzStatus(address=uplink['ip'],
password=uplink['password'])
if fc.is_connected:
self.status = 'UP'
self.__configuration.set_env_var('_last_success_' + uplink['identifier'],
self.date + ' ' + self.time)
else:
self.status = 'DOWN'
self.__configuration.set_env_var('_last_fail_' + uplink['identifier'],
self.date + ' ' + self.time)
logger.info(str(uplink['provider'] + ' (uplink IP: ' + fc.external_ip + ') ' +
logger.info(str(uplink['provider'] + ' (IP: ' + fc.external_ip + ') ' +
self.status))
model = Model(self.configuration)
model = Model(self.__configuration)
model.set_date(self.date)
model.set_external_ip(fc.external_ip)
model.set_external_ipv6(fc.external_ipv6)
@ -86,7 +93,7 @@ class Uplink(Daemon):
model.set_timestamp(timestamp)
model.set_uptime(fc.connection_uptime)
database = Database(self.configuration)
database = Database(self.__configuration)
database.write_model_to_db(model)
except Exception as err:
@ -94,14 +101,14 @@ class Uplink(Daemon):
logger.error(str(message.format(err)))
def run(self):
if self.configuration.get_env_var('_server'):
from uplink.server_cherrypy import Server
s = Server(self.configuration)
if self.__configuration.get_env_var('_httpserver'):
from uplink.httpserver import HTTPServer
s = HTTPServer(self.__configuration)
st = Thread(target=s.run_server, daemon=True)
st.start()
while True:
for i in range(len(self.configuration.get_uplinks())):
for i in range(len(self.__configuration.get_uplinks())):
ut = Thread(target=self.fetch_data, daemon=True, args=(i,))
ut.start()
time.sleep(self.configuration.get_interval())
time.sleep(self.__configuration.get_interval())