args cleanup and full configuration in config file possible now. lot of refactoring and tons of cleanups

README.md

@@ -1,108 +1,3 @@
-# uplink - is not some program expecting uplinks to work!
+# uplink -is a tool to monitor the uplink status of AVM FRITZ!Box Cable and DSL based routers.
 
-## **This README file is not up-to-date! It will be fixed soon to explain the installation and the usage of uplink.**
+Go to [https://hanez.org/uplink.html](https://hanez.org/uplink.html) to view the latest documentation.
-
-uplink is a tool to monitor the uplink status of AVM FRITZ!Box Cable and DSL based routers. It uses the TR-064 protocol over UPnP.
-
-## Features
-
-For now, I can say that uplink can monitor the status of you FRITZ!Box uplinks. It can log to a file and writes results to a MariaDB/MySQL database. With some editing of the code you can you use SQLite too but this feature is not active at the moment. This will change in the future. I switched to MariaDB because I run uplink on a RaspberryPi and are evaluating results on my workstation. I am working on a Gtk+ frontend to generate statistics...
-
-## Requirements
-
- - A Linux based operating system (I don't test any other OS's but Raspbian is fine and a good target)
- - Git (only for installing from Git repository)
- - Python3 (running for me using 3.7.3, 3.8.5, 3.9.7 and 3.10.1 on Linux) 
- - fritzconnection >= 1.5.0 (included as submodule in the Git repository)
- - pymysql >= 0.10.1
- - sqlalchemy = 1.4.27 (not for now)
- - urwid = 2.1.2 (not for now)
-
-Just install the 3rd party dependencies using `pip install $PACKAGE` or your OS package manager
-## Installation
-
-    git clone https://git.unixpeople.org/hanez/uplink.git
-    cd uplink
-    # To get a local copy of fritzconnection you need to get the submodules:
-    git submodule update --init --recursive
-    cp config.example.json config.json
-
-### Configuration
-
-Edit config.json to your needs.
-
-#### Variables
-
- - interval: The polling interval
- - uplinks: List of devices you want to poll
-   - provider: Custom name of the uplink provider
-   - ip: The IP of the router device
-   - password: The password of the router device
-
-#### Example
-
-    {
-    "interval": 60,
-    "database_type": "mysql", (NOT USED)
-    "database":  "uplink",
-    "database_host": "127.0.0.1",
-    "database_user": "USER",
-    "database_password": "PASSWORD",
-    "cron": false, (NOT USED)
-    "daemon": false, (NOT USED)
-        "uplinks": [
-            { "provider": "Cable Provider", "ip": "192.168.0.1", "password": "1234" },
-            { "provider": "DSL Provider", "ip": "192.168.1.1", "password": "1234" }
-        ]
-    }
-
-### Create Database
-
-    TODO
-   For now there is only the file uplink.sql which will create the tables in your MariaDB/MySQL database.
-
-   The file is under files/uplink.sql
-
-## Usage
-
-    ./uplink [-c | -d | -f] ./config.json 
-
-### Help
-
-    ./uplink --help
-
-   ```
-   usage: uplink [-h] [-c | -d | -f] [-i INTERVAL] [--version] CONFIGFILE
-   
-   uplink is a tool to monitor the link status of AVM FRITZ!Box Cable and DSL based routers.
-   
-   positional arguments:
-   CONFIGFILE            the configfile to use
-   
-   options:
-   -h, --help            show this help message and exit
-   -c, --cron            cron mode executes the script only once without loop (default: false)
-   -d, --daemon          run as native daemon (default: false)
-   -f, --foreground      run looped in foreground (default: false)
-   -i INTERVAL, --interval INTERVAL
-   poll interval in seconds. this overrides config file settings. (default: 60)
-   --version             show program's version number and exit
-   ```
-
-## View collected data
-
-I actually use "[DBeaver](https://dbeaver.io/)" for taking a look at the data uplink is collecting.
-
-There will be a Gtk+ frontend to uplink at some time but this project is at a very early stage of development, so I want to write the collector first. Even support for other SQL databases is in planning in conjunction with the Gtk+ frontend. I use MariaDB only because I can move fast-forward.
-
-## Planned features
-
-- A lot... :)
-- Make all config vars as ARGS and vice versa. ARGS have higher priority. Chain: default -> config -> ARGS.
-- Bring back SQLite support.
-- Switch to ORM (peewee, sqlalchemy?) to support PostgreSQL, MariaDB/MySQL, SQLite and maybe more databases.
-- Gtk+/urwid frontend for visualizing the collected data. wxGlade?
-- A tool to generate reports for showing to your uplink provider.
-- Always keep platform independence in mind but not if uplink looses nice features on Linux.
-- Implement a speedtest feature to regularly run speedtest but independent to uptime checks with different interval.
-- A small embedded webserver to view what is going on.

bin/uplink

@@ -71,37 +71,18 @@ def parse_args():
     mode.add_argument('-f', '--foreground', default=False, dest='foreground', action='store_true',
                       help='run looped in foreground (default: false)')
 
-    parser.add_argument('--server', default=False, dest='server', action='store_true',
+    parser.add_argument('--httpserver', default=False, dest='httpserver', action='store_true',
                         help='start http server for status information and statistics')
 
     parser.add_argument('-i', '--interval', type=int, help='poll interval in seconds. this '
-                        'overrides config file settings. (default: 60)')
+                        'overrides config file settings (default: 60)')
 
-    parser.add_argument('-l', '--logfile', metavar='LOGFILE', help='logfile to use')
+    parser.add_argument('-l', '--logfile', metavar='LOGFILE', help='logfile to use. this '
-
+                                                                   'overrides config file settings')
-    parser.add_argument('-n', '--logcount', default=5, type=int,
-                        help='maximum number of logfiles in rotation (default: 5)')
-
-    parser.add_argument('-m', '--logsize', default=10485760, type=int,
-                        help='maximum logfile size in bytes (default: 1048576)')
 
     parser.add_argument('-s', '--stdout', default=False, dest='stdout', action='store_true',
                         help='log to stdout')
 
-    output = parser.add_mutually_exclusive_group()
-    output.add_argument('-q', '--quiet', action='store_const', dest='loglevel',
-                        const=logging.ERROR, help='output only errors')
-
-    output.add_argument('-w', '--warning', action='store_const', dest='loglevel',
-                        const=logging.WARNING, help='output warnings')
-
-    output.add_argument('-v', '--verbose', action='store_const', dest='loglevel',
-                        const=logging.INFO, help='output info messages')
-
-    output.add_argument('-e', '--debug', action='store_const', dest='loglevel',
-                        const=logging.DEBUG, help='output debug messages')
-    output.set_defaults(loglevel=logging.ERROR)
-
     parser.add_argument('--version', action='version', version='%(prog)s ' + str(__version__))
 
     return parser.parse_args()
@@ -112,25 +93,6 @@ def main():
 
     root_logger = logging.getLogger()
 
-    if args.logfile:
-        logfile = os.path.expanduser(args.logfile)
-        if not os.path.exists(os.path.dirname(logfile)):
-            os.makedirs(os.path.dirname(logfile))
-
-        formatter1 = logging.Formatter('%(asctime)s:%(levelname)s:%(name)s:%(message)s')
-        handler1 = logging.handlers.RotatingFileHandler(args.logfile, maxBytes=args.logsize,
-                                                        backupCount=args.logcount)
-        handler1.setFormatter(formatter1)
-        root_logger.addHandler(handler1)
-
-    if args.stdout:
-        formatter2 = logging.Formatter('[%(asctime)s] [%(levelname)s] %(message)s')
-        handler2 = logging.StreamHandler(sys.stdout)
-        handler2.setFormatter(formatter2)
-        root_logger.addHandler(handler2)
-
-    root_logger.setLevel(args.loglevel)
-
     try:
         with open(args.configuration_file, 'r', encoding='utf-8') as configuration_file:
             configuration_data = configuration_file.read()
@@ -147,9 +109,39 @@ def main():
     if args.interval:
         configuration.set_interval(args.interval)
 
-    configuration.set_env_var('_server', False)
+    if args.logfile:
-    if args.server:
+        configuration.set_log_file(args.logfile)
-        configuration.set_env_var('_server', True)
+
+    if args.logfile or configuration.get_log_file():
+        log_file = os.path.expanduser(configuration.get_log_file())
+        if not os.path.exists(os.path.dirname(log_file)):
+            os.makedirs(os.path.dirname(log_file))
+
+        formatter1 = logging.Formatter('%(asctime)s:%(levelname)s:%(name)s:%(message)s')
+        handler1 = logging.handlers.RotatingFileHandler(log_file,
+                                                        maxBytes=configuration.get_log_size(),
+                                                        backupCount=configuration.get_log_count())
+        handler1.setFormatter(formatter1)
+        root_logger.addHandler(handler1)
+
+    if args.stdout:
+        formatter2 = logging.Formatter('[%(asctime)s] [%(levelname)s] %(message)s')
+        handler2 = logging.StreamHandler(sys.stdout)
+        handler2.setFormatter(formatter2)
+        root_logger.addHandler(handler2)
+
+    # errors will always show up even when no log_level is set!
+    root_logger.setLevel(logging.ERROR)
+    log_level = configuration.get_log_level()
+    if log_level == "warning":
+        root_logger.setLevel(logging.WARNING)
+    elif log_level == "verbose":
+        root_logger.setLevel(logging.INFO)
+    elif log_level == "debug":
+        root_logger.setLevel(logging.DEBUG)
+
+    if args.httpserver or configuration.get_httpserver():
+        configuration.set_env_var('_httpserver', True)
 
     try:
         u = Uplink(configuration.get_pid_file(), configuration)
@@ -171,9 +163,9 @@ def main():
             u.start()
     elif args.foreground:
         from threading import Thread
-        if configuration.get_env_var('_server'):
+        if configuration.get_env_var('_httpserver'):
-            from uplink.server_cherrypy import Server
+            from uplink.httpserver import HTTPServer
-            s = Server(configuration)
+            s = HTTPServer(configuration)
             st = Thread(target=s.run_server, daemon=True)
             st.start()
 

etc/config.example.json

@@ -1,14 +1,17 @@
 {
-    "interval": 60,
-    "database_name": "uplink",
     "database_host": "127.0.0.1",
-    "database_user": "USER",
+    "database_name": "uplink",
-    "database_password": "PASSWORD",
+    "database_password": "uplink",
+    "database_type": "mysql",
+    "database_user": "uplink",
     "httpserver_host": "0.0.0.0",
-    "httpserver_port": 8080,
+    "httpserver_port": 4000,
-    "httpserver_framework": "cherrypy",
+    "interval": 30,
-    "mode": "cron",
     "log_level": "info",
+    "log_count": 5,
+    "log_size": 0,
+    "pid_file": "/var/run/user/1000/uplink.pid",
+    "run_mode": "cron",
     "uplinks": [
         {
             "provider": "Cable Provider",
@@ -21,4 +24,4 @@
             "password": "1234"
         }
     ]
-}
+}

uplink/configuration.py

@@ -26,27 +26,27 @@ logger = getLogger(__name__)
 class Configuration:
 
     def __init__(self, configuration):
-
         self.__configuration = configuration
+
         self.__database_host = None
         self.__database_name = None
         self.__database_password = None
         self.__database_user = None
-        self.__httpserver_framework = None
+        # environment vars which can be set dynamically at runtime.
+        self.__env = {}
+        self.__httpserver = False
         self.__httpserver_host = '0.0.0.0'
         self.__httpserver_port = 8080
-        self.__interval = '60'
+        self.__interval = 60
-        self.__log_count = 5
+        self.__log_count = 1
         self.__log_file = None
-        self.__log_level = 'info'
+        self.__log_level = None
-        self.__log_size = 10485760,
+        self.__log_size = 0
         self.__pid_file = '/tmp/uplink.pid'
-        self.__run_mode = None
+        self.__run_mode = 'cron'
         self.__uplinks = None
 
-        self.__env = {}
+        if 'database_host' in self.__configuration:
-
-        if'database_host' in self.__configuration:
             self.__database_host = self.__configuration['database_host']
         else:
             raise Exception('database_host required!')
@@ -66,8 +66,8 @@ class Configuration:
         else:
             raise Exception('database_user required!')
 
-        if 'httpserver_framework' in self.__configuration:
+        if 'httpserver' in self.__configuration:
-            self.__httpserver_framework = self.__configuration['httpserver_framework']
+            self.__httpserver = self.__configuration['httpserver']
 
         if 'httpserver_host' in self.__configuration:
             self.__httpserver_host = self.__configuration['httpserver_host']
@@ -78,9 +78,21 @@ class Configuration:
         if 'interval' in self.__configuration:
             self.__interval = self.__configuration['interval']
 
+        if 'log_count' in self.__configuration:
+            self.__log_count = self.__configuration['log_count']
+
+        if 'log_file' in self.__configuration:
+            self.__log_file = self.__configuration['log_file']
+
         if 'log_level' in self.__configuration:
             self.__log_level = self.__configuration['log_level']
 
+        if 'log_size' in self.__configuration:
+            self.__log_size = self.__configuration['log_size']
+
+        if 'pid_file' in self.__configuration:
+            self.__pid_file = self.__configuration['pid_file']
+
         if 'run_mode' in self.__configuration:
             self.__run_mode = self.__configuration['run_mode']
 
@@ -89,6 +101,7 @@ class Configuration:
         else:
             raise Exception('uplinks required!')
 
+    # get methods
     def get_database_host(self):
         return self.__database_host
 
@@ -105,13 +118,11 @@ class Configuration:
         return self.__env
 
     def get_env_var(self, name):
-        return self.__env[name]
+        if name in self.__env:
+            return self.__env[name]
 
-    def set_env_var(self, name, value):
+    def get_httpserver(self):
-        self.__env[name] = value
+        return self.__httpserver
-
-    def get_httpserver_framework(self):
-        return self.__httpserver_framework
 
     def get_httpserver_host(self):
         return self.__httpserver_host
@@ -122,12 +133,18 @@ class Configuration:
     def get_interval(self):
         return self.__interval
 
-    def set_interval(self, interval):
+    def get_log_file(self):
-        self.__interval = interval
+        return self.__log_file
+
+    def get_log_count(self):
+        return self.__log_count
 
     def get_log_level(self):
         return self.__log_level
 
+    def get_log_size(self):
+        return self.__log_size
+
     def get_pid_file(self):
         return self.__pid_file
 
@@ -139,3 +156,16 @@ class Configuration:
 
     def get_uplinks(self):
         return self.__uplinks
+
+    # set methods
+    def set_env_var(self, name, value):
+        self.__env[name] = value
+
+    def set_httpserver(self, value):
+        self.__httpserver = value
+
+    def set_interval(self, interval):
+        self.__interval = interval
+
+    def set_log_file(self, value):
+        self.__log_file = value

uplink/csv.py

@@ -23,10 +23,10 @@ from logging import getLogger
 logger = getLogger(__name__)
 
 
-class Csv:
+class CSV:
 
     def __init__(self, configuration):
-        self.configuration = configuration
+        self.__configuration = configuration
 
     def write_model_to_csv(self):
         return

uplink/httpserver.py

@@ -29,7 +29,7 @@ from logging import getLogger
 logger = getLogger(__name__)
 
 
-class Server:
+class HTTPServer:
 
     def __init__(self, configuration):
         self.__configuration = configuration

uplink/model.py

@@ -36,6 +36,7 @@ class Model:
         self.__internal_ip = None
         self.__is_connected = None
         self.__is_linked = None
+        # TODO: rename message to status
         self.__message = None
         self.__model_name = None
         self.__provider = None

uplink/notification.py

@@ -0,0 +1,29 @@
+# Copyright (c) 2022 Johannes Findeisen <you@hanez.org>
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is furnished
+# to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice (including the next
+# paragraph) shall be included in all copies or substantial portions of the
+# Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
+# FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS
+# OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+# WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF
+# OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+from logging import getLogger
+
+logger = getLogger(__name__)
+
+
+class Notification:
+
+    def __init__(self, configuration):
+        self.__configuration = configuration

uplink/speedtest.py

@@ -28,7 +28,8 @@ logger = getLogger(__name__)
 class Speedtest:
 
     def __init__(self, configuration):
-        self.configuration = configuration
+        self.__configuration = configuration
+
         self.download = None
         self.upload = None
 

uplink/uplink.py

@@ -18,10 +18,9 @@
 # WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF
 # OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
 
-# TODO: Switch all SQL stuff to SQLAlchemy using the uplink.database Database class
 # TODO: Make the use of a database optional and only output to stdout and/or in a logfile (use rich
 #  for colorizing the std output (https://github.com/Textualize/rich)); Output to a CSV file should
-#  be optional too.
+#  be optional too. Also reinvent SQLite as database backend.
 
 import calendar
 import socket
@@ -33,6 +32,7 @@ from threading import Thread
 from uplink.daemon import Daemon
 from uplink.database import Database
 from uplink.model import Model
+from uplink.notification import Notification
 
 logger = getLogger(__name__)
 
@@ -41,7 +41,8 @@ class Uplink(Daemon):
 
     def __init__(self, pid_file, configuration):
         super().__init__(pid_file)
-        self.configuration = configuration
+        self.__configuration = configuration
+
         self.date = None
         self.time = None
         self.status = None
@@ -52,19 +53,25 @@ class Uplink(Daemon):
         self.date = time.strftime('%Y-%m-%d', local_time)
         self.time = time.strftime('%H:%M:%S', local_time)
 
-        uplink = self.configuration.get_uplink(i)
+        self.__configuration.set_env_var('_last_run', self.date + ' ' + self.time)
+
+        uplink = self.__configuration.get_uplink(i)
         try:
             fc = FritzStatus(address=uplink['ip'],
                              password=uplink['password'])
             if fc.is_connected:
                 self.status = 'UP'
+                self.__configuration.set_env_var('_last_success_' + uplink['identifier'],
+                                                 self.date + ' ' + self.time)
             else:
                 self.status = 'DOWN'
+                self.__configuration.set_env_var('_last_fail_' + uplink['identifier'],
+                                                 self.date + ' ' + self.time)
 
-            logger.info(str(uplink['provider'] + ' (uplink IP: ' + fc.external_ip + ') ' +
+            logger.info(str(uplink['provider'] + ' (IP: ' + fc.external_ip + ') ' +
                             self.status))
 
-            model = Model(self.configuration)
+            model = Model(self.__configuration)
             model.set_date(self.date)
             model.set_external_ip(fc.external_ip)
             model.set_external_ipv6(fc.external_ipv6)
@@ -86,7 +93,7 @@ class Uplink(Daemon):
             model.set_timestamp(timestamp)
             model.set_uptime(fc.connection_uptime)
 
-            database = Database(self.configuration)
+            database = Database(self.__configuration)
             database.write_model_to_db(model)
 
         except Exception as err:
@@ -94,14 +101,14 @@ class Uplink(Daemon):
             logger.error(str(message.format(err)))
 
     def run(self):
-        if self.configuration.get_env_var('_server'):
+        if self.__configuration.get_env_var('_httpserver'):
-            from uplink.server_cherrypy import Server
+            from uplink.httpserver import HTTPServer
-            s = Server(self.configuration)
+            s = HTTPServer(self.__configuration)
             st = Thread(target=s.run_server, daemon=True)
             st.start()
 
         while True:
-            for i in range(len(self.configuration.get_uplinks())):
+            for i in range(len(self.__configuration.get_uplinks())):
                 ut = Thread(target=self.fetch_data, daemon=True, args=(i,))
                 ut.start()
-            time.sleep(self.configuration.get_interval())
+            time.sleep(self.__configuration.get_interval())