Bump version to 1.3.1. Skipping 1.3 as a tag already exists

Fix ConfigurationProfiles

Dockerfile

@@ -38,12 +38,15 @@ RUN apt update \
 # Build libimobiledevice
 # ----------------------
 RUN git clone https://github.com/libimobiledevice/libplist \
+  && git clone https://github.com/libimobiledevice/libimobiledevice-glue \
   && git clone https://github.com/libimobiledevice/libusbmuxd \
   && git clone https://github.com/libimobiledevice/libimobiledevice \
   && git clone https://github.com/libimobiledevice/usbmuxd \
 
   && cd libplist && ./autogen.sh && make && make install && ldconfig \
 
+  && cd ../libimobiledevice-glue && PKG_CONFIG_PATH=/usr/local/lib/pkgconfig ./autogen.sh --prefix=/usr && make && make install && ldconfig \
+
   && cd ../libusbmuxd && PKG_CONFIG_PATH=/usr/local/lib/pkgconfig ./autogen.sh && make && make install && ldconfig \
 
   && cd ../libimobiledevice && PKG_CONFIG_PATH=/usr/local/lib/pkgconfig ./autogen.sh --enable-debug && make && make install && ldconfig \
@@ -51,7 +54,7 @@ RUN git clone https://github.com/libimobiledevice/libplist \
   && cd ../usbmuxd && PKG_CONFIG_PATH=/usr/local/lib/pkgconfig ./autogen.sh --prefix=/usr --sysconfdir=/etc --localstatedir=/var --runstatedir=/run && make && make install \
 
   # Clean up.
-  && cd .. && rm -rf libplist libusbmuxd libimobiledevice usbmuxd
+  && cd .. && rm -rf libplist libimobiledevice-glue libusbmuxd libimobiledevice usbmuxd
 
 # Installing MVT
 # --------------

README.md

@@ -15,15 +15,15 @@ It has been developed and released by the [Amnesty International Security Lab](h
 
 ## Installation
 
-MVT can be installed from sources or from [PyPi](https://pypi.org/project/mvt/) (you will need some dependencies, check the [documentation](https://docs.mvt.re/en/latest/install.html)):
+MVT can be installed from sources or from [PyPi](https://pypi.org/project/mvt/) (you will need some dependencies, check the [documentation](https://docs.mvt.re/en/latest/install/)):
 
 ```
 pip3 install mvt
 ```
 
-Alternatively, you can decide to run MVT and all relevant tools through a [Docker container](https://docs.mvt.re/en/latest/docker.html).
+Alternatively, you can decide to run MVT and all relevant tools through a [Docker container](https://docs.mvt.re/en/latest/docker/).
 
-**Please note:** MVT is best run on Linux or Mac systems. [It does not currently support running natively on Windows.](https://docs.mvt.re/en/latest/install.html#mvt-on-windows)
+**Please note:** MVT is best run on Linux or Mac systems. [It does not currently support running natively on Windows.](https://docs.mvt.re/en/latest/install/#mvt-on-windows)
 
 ## Usage
 
@@ -31,4 +31,4 @@ MVT provides two commands `mvt-ios` and `mvt-android`. [Check out the documentat
 
 ## License
 
-The purpose of MVT is to facilitate the ***consensual forensic analysis*** of devices of those who might be targets of sophisticated mobile spyware attacks, especially members of civil society and marginalized communities. We do not want MVT to enable privacy violations of non-consenting individuals.  In order to achieve this, MVT is released under its own license. [Read more here.](https://docs.mvt.re/en/latest/license.html)
+The purpose of MVT is to facilitate the ***consensual forensic analysis*** of devices of those who might be targets of sophisticated mobile spyware attacks, especially members of civil society and marginalized communities. We do not want MVT to enable privacy violations of non-consenting individuals.  In order to achieve this, MVT is released under its own license. [Read more here.](https://docs.mvt.re/en/latest/license/)

docs/android/backup.md

@@ -22,7 +22,7 @@ adb backup -all
 
 ## Unpack the backup
 
-In order to reliable unpack th [Android Backup Extractor (ABE)](https://github.com/nelenkov/android-backup-extractor) to convert it to a readable file format. Make sure that java is installed on your system and use the following command:
+In order to unpack the backup, use [Android Backup Extractor (ABE)](https://github.com/nelenkov/android-backup-extractor) to convert it to a readable file format. Make sure that java is installed on your system and use the following command:
 
 ```bash
 java -jar ~/path/to/abe.jar unpack backup.ab backup.tar
@@ -31,6 +31,8 @@ tar xvf backup.tar
 
 If the backup is encrypted, ABE will prompt you to enter the password.
 
+Alternatively, [ab-decrypt](https://github.com/joernheissler/ab-decrypt) can be used for that purpose.
+
 ## Check the backup
 
 You can then extract SMSs containing links with MVT:

docs/android/methodology.md

@@ -8,7 +8,7 @@ However, not all is lost.
 
 Because malware attacks over Android typically take the form of malicious or backdoored apps, the very first thing you might want to do is to extract and verify all installed Android packages and triage quickly if there are any which stand out as malicious or which might be atypical.
 
-While it is out of the scope of this documentation to dwell into details on how to analyze Android apps, MVT does allow to easily and automatically extract information about installed apps, download copies of them, and quickly lookup services such as [VirusTotal](https://www.virustotal.com) or [Koodous](https://www.koodous.com) which might quickly indicate known bad apps.
+While it is out of the scope of this documentation to dwell into details on how to analyze Android apps, MVT does allow to easily and automatically extract information about installed apps, download copies of them, and quickly lookup services such as [VirusTotal](https://www.virustotal.com) or [Koodous](https://koodous.com) which might quickly indicate known bad apps.
 
 
 ## Check the device over Android Debug Bridge

docs/iocs.md

@@ -28,9 +28,16 @@ The `--iocs` option can be invoked multiple times to let MVT import multiple STI
 mvt-ios check-backup --iocs ~/iocs/malware1.stix --iocs ~/iocs/malware2.stix2 /path/to/backup
 ```
 
+It is also possible to load STIX2 files automatically from the environment variable `MVT_STIX2`:
+
+```bash
+export MVT_STIX2="/home/user/IOC1.stix2:/home/user/IOC2.stix2"
+```
+
 ## Known repositories of STIX2 IOCs
 
 - The [Amnesty International investigations repository](https://github.com/AmnestyTech/investigations) contains STIX-formatted IOCs for:
     - [Pegasus](https://en.wikipedia.org/wiki/Pegasus_(spyware)) ([STIX2](https://raw.githubusercontent.com/AmnestyTech/investigations/master/2021-07-18_nso/pegasus.stix2))
+- [This repository](https://github.com/Te-k/stalkerware-indicators) contains IOCs for Android stalkerware including [a STIX MVT-compatible file](https://github.com/Te-k/stalkerware-indicators/blob/master/stalkerware.stix2).
 
 Please [open an issue](https://github.com/mvt-project/mvt/issues/) to suggest new sources of STIX-formatted IOCs.

docs/requirements.txt

@@ -1,4 +1,4 @@
-mkdocs==1.2.1
+mkdocs==1.2.3
 mkdocs-autorefs
 mkdocs-material
 mkdocs-material-extensions

mvt/android/cli.py

@@ -9,8 +9,11 @@ import os
 import click
 from rich.logging import RichHandler
 
-from mvt.common.help import *
+from mvt.common.help import HELP_MSG_MODULE, HELP_MSG_IOC
+from mvt.common.help import HELP_MSG_FAST, HELP_MSG_OUTPUT, HELP_MSG_LIST_MODULES
+from mvt.common.help import HELP_MSG_SERIAL
 from mvt.common.indicators import Indicators, IndicatorsFileBadFormat
+from mvt.common.logo import logo
 from mvt.common.module import run_module, save_timeline
 
 from .download_apks import DownloadAPKs
@@ -25,11 +28,20 @@ logging.basicConfig(level="INFO", format=LOG_FORMAT, handlers=[
     RichHandler(show_path=False, log_time_format="%X")])
 log = logging.getLogger(__name__)
 
+
 #==============================================================================
 # Main
 #==============================================================================
 @click.group(invoke_without_command=False)
 def cli():
+    logo()
+
+
+#==============================================================================
+# Command: version
+#==============================================================================
+@cli.command("version", help="Show the currently installed version of MVT")
+def version():
     return
 
 
@@ -95,10 +107,11 @@ def download_apks(ctx, all_apks, virustotal, koodous, all_checks, output, from_f
               default=[], help=HELP_MSG_IOC)
 @click.option("--output", "-o", type=click.Path(exists=False),
               help=HELP_MSG_OUTPUT)
+@click.option("--fast", "-f", is_flag=True, help=HELP_MSG_FAST)
 @click.option("--list-modules", "-l", is_flag=True, help=HELP_MSG_LIST_MODULES)
 @click.option("--module", "-m", help=HELP_MSG_MODULE)
 @click.pass_context
-def check_adb(ctx, iocs, output, list_modules, module, serial):
+def check_adb(ctx, iocs, output, fast, list_modules, module, serial):
     if list_modules:
         log.info("Following is the list of available check-adb modules:")
         for adb_module in ADB_MODULES:
@@ -130,7 +143,8 @@ def check_adb(ctx, iocs, output, list_modules, module, serial):
         if module and adb_module.__name__ != module:
             continue
 
-        m = adb_module(output_folder=output, log=logging.getLogger(adb_module.__module__))
+        m = adb_module(output_folder=output, fast_mode=fast,
+                       log=logging.getLogger(adb_module.__module__))
         if serial:
             m.serial = serial
 
@@ -182,7 +196,7 @@ def check_backup(ctx, iocs, output, backup_path, serial):
         log.critical("The path you specified is a not a folder!")
 
         if os.path.basename(backup_path) == "backup.ab":
-            log.info("You can use ABE (https://github.com/nelenkov/android-backup-extractor) " \
+            log.info("You can use ABE (https://github.com/nelenkov/android-backup-extractor) "
                      "to extract 'backup.ab' files!")
         ctx.exit(1)
 
@@ -193,7 +207,7 @@ def check_backup(ctx, iocs, output, backup_path, serial):
         if serial:
             m.serial = serial
 
-        if iocs:
+        if len(indicators.ioc_count) > 0:
             indicators.log = m.log
             m.indicators = indicators
 

mvt/android/download_apks.py

@@ -7,17 +7,16 @@ import json
 import logging
 import os
 
-import pkg_resources
 from tqdm import tqdm
 
 from mvt.common.module import InsufficientPrivileges
-from mvt.common.utils import get_sha256_from_file_path
 
 from .modules.adb.base import AndroidExtraction
 from .modules.adb.packages import Packages
 
 log = logging.getLogger(__name__)
 
+
 # TODO: Would be better to replace tqdm with rich.progress to reduce
 #       the number of dependencies. Need to investigate whether
 #       it's possible to have a similar callback system.
@@ -32,7 +31,10 @@ class PullProgress(tqdm):
 
 class DownloadAPKs(AndroidExtraction):
     """DownloadAPKs is the main class operating the download of APKs
-    from the device."""
+    from the device.
+
+
+    """
 
     def __init__(self, output_folder=None, all_apks=False, log=None,
                  packages=None):
@@ -51,7 +53,9 @@ class DownloadAPKs(AndroidExtraction):
     @classmethod
     def from_json(cls, json_path):
         """Initialize this class from an existing apks.json file.
+
         :param json_path: Path to the apks.json file to parse.
+
         """
         with open(json_path, "r") as handle:
             packages = json.load(handle)
@@ -59,9 +63,11 @@ class DownloadAPKs(AndroidExtraction):
 
     def pull_package_file(self, package_name, remote_path):
         """Pull files related to specific package from the device.
+
         :param package_name: Name of the package to download
         :param remote_path: Path to the file to download
         :returns: Path to the local copy
+
         """
         log.info("Downloading %s ...", remote_path)
 
@@ -101,6 +107,8 @@ class DownloadAPKs(AndroidExtraction):
     def get_packages(self):
         """Use the Packages adb module to retrieve the list of packages.
         We reuse the same extraction logic to then download the APKs.
+
+
         """
         self.log.info("Retrieving list of installed packages...")
 
@@ -111,8 +119,7 @@ class DownloadAPKs(AndroidExtraction):
         self.packages = m.results
 
     def pull_packages(self):
-        """Download all files of all selected packages from the device.
-        """
+        """Download all files of all selected packages from the device."""
         log.info("Starting extraction of installed APKs at folder %s", self.output_folder)
 
         if not os.path.exists(self.output_folder):
@@ -131,7 +138,7 @@ class DownloadAPKs(AndroidExtraction):
                     packages_selection.append(package)
 
             log.info("Selected only %d packages which are not marked as system",
-                len(packages_selection))
+                     len(packages_selection))
 
         if len(packages_selection) == 0:
             log.info("No packages were selected for download")
@@ -150,50 +157,27 @@ class DownloadAPKs(AndroidExtraction):
             log.info("[%d/%d] Package: %s", counter, len(packages_selection),
                      package["package_name"])
 
-            # Get the file path for the specific package.
-            try:
-                output = self._adb_command(f"pm path {package['package_name']}")
-                output = output.strip().replace("package:", "")
-                if not output:
-                    continue
-            except Exception as e:
-                log.exception("Failed to get path of package %s: %s",
-                              package["package_name"], e)
-                self._adb_reconnect()
-                continue
-
             # Sometimes the package path contains multiple lines for multiple apks.
             # We loop through each line and download each file.
-            for path in output.split("\n"):
-                device_path = path.strip()
-                file_path = self.pull_package_file(package["package_name"],
-                                                   device_path)
-                if not file_path:
+            for package_file in package["files"]:
+                device_path = package_file["path"]
+                local_path = self.pull_package_file(package["package_name"],
+                                                    device_path)
+                if not local_path:
                     continue
 
-                file_info = {
-                    "path": device_path,
-                    "local_name": file_path,
-                    "sha256": get_sha256_from_file_path(file_path),
-                }
-
-                if "files" not in package:
-                    package["files"] = [file_info,]
-                else:
-                    package["files"].append(file_info)
+                package_file["local_path"] = local_path
 
         log.info("Download of selected packages completed")
 
     def save_json(self):
-        """Save the results to the package.json file.
-        """
+        """Save the results to the package.json file."""
         json_path = os.path.join(self.output_folder, "apks.json")
         with open(json_path, "w") as handle:
             json.dump(self.packages, handle, indent=4)
 
     def run(self):
-        """Run all steps of fetch-apk.
-        """
+        """Run all steps of fetch-apk."""
         self.get_packages()
         self._adb_connect()
         self.pull_packages()

mvt/android/lookups/koodous.py

@@ -13,6 +13,7 @@ from rich.text import Text
 
 log = logging.getLogger(__name__)
 
+
 def koodous_lookup(packages):
     log.info("Looking up all extracted files on Koodous (www.koodous.com)")
     log.info("This might take a while...")
@@ -32,7 +33,7 @@ def koodous_lookup(packages):
             res = requests.get(url)
             report = res.json()
 
-            row = [package["package_name"], file["local_name"]]
+            row = [package["package_name"], file["path"]]
 
             if "package_name" in report:
                 trusted = "no"

mvt/android/lookups/virustotal.py

@@ -13,6 +13,7 @@ from rich.text import Text
 
 log = logging.getLogger(__name__)
 
+
 def get_virustotal_report(hashes):
     apikey = "233f22e200ca5822bd91103043ccac138b910db79f29af5616a9afe8b6f215ad"
     url = f"https://www.virustotal.com/partners/sysinternals/file-reports?apikey={apikey}"
@@ -36,6 +37,7 @@ def get_virustotal_report(hashes):
         log.error("Unexpected response from VirusTotal: %s", res.status_code)
         return None
 
+
 def virustotal_lookup(packages):
     log.info("Looking up all extracted files on VirusTotal (www.virustotal.com)")
 
@@ -48,6 +50,7 @@ def virustotal_lookup(packages):
     total_unique_hashes = len(unique_hashes)
 
     detections = {}
+
     def virustotal_query(batch):
         report = get_virustotal_report(batch)
         if not report:
@@ -75,7 +78,7 @@ def virustotal_lookup(packages):
 
     for package in packages:
         for file in package.get("files", []):
-            row = [package["package_name"], file["local_name"]]
+            row = [package["package_name"], file["path"]]
 
             if file["sha256"] in detections:
                 detection = detections[file["sha256"]]

mvt/android/modules/adb/__init__.py

@@ -5,8 +5,12 @@
 
 from .chrome_history import ChromeHistory
 from .dumpsys_batterystats import DumpsysBatterystats
+from .dumpsys_full import DumpsysFull
 from .dumpsys_packages import DumpsysPackages
 from .dumpsys_procstats import DumpsysProcstats
+from .dumpsys_receivers import DumpsysReceivers
+from .files import Files
+from .logcat import Logcat
 from .packages import Packages
 from .processes import Processes
 from .rootbinaries import RootBinaries
@@ -15,4 +19,5 @@ from .whatsapp import Whatsapp
 
 ADB_MODULES = [ChromeHistory, SMS, Whatsapp, Processes,
                DumpsysBatterystats, DumpsysProcstats,
-               DumpsysPackages, Packages, RootBinaries]
+               DumpsysPackages, DumpsysReceivers, DumpsysFull,
+               Packages, RootBinaries, Logcat, Files]

mvt/android/modules/adb/base.py

@@ -25,6 +25,7 @@ log = logging.getLogger(__name__)
 ADB_KEY_PATH = os.path.expanduser("~/.android/adbkey")
 ADB_PUB_KEY_PATH = os.path.expanduser("~/.android/adbkey.pub")
 
+
 class AndroidExtraction(MVTModule):
     """This class provides a base for all Android extraction modules."""
 
@@ -37,9 +38,12 @@ class AndroidExtraction(MVTModule):
         self.device = None
         self.serial = None
 
-    def _adb_check_keys(self):
-        """Make sure Android adb keys exist.
-        """
+    @staticmethod
+    def _adb_check_keys():
+        """Make sure Android adb keys exist."""
+        if not os.path.isdir(os.path.dirname(ADB_KEY_PATH)):
+            os.makedirs(os.path.dirname(ADB_KEY_PATH))
+
         if not os.path.exists(ADB_KEY_PATH):
             keygen(ADB_KEY_PATH)
 
@@ -47,14 +51,16 @@ class AndroidExtraction(MVTModule):
             write_public_keyfile(ADB_KEY_PATH, ADB_PUB_KEY_PATH)
 
     def _adb_connect(self):
-        """Connect to the device over adb.
-        """
+        """Connect to the device over adb."""
         self._adb_check_keys()
 
         with open(ADB_KEY_PATH, "rb") as handle:
             priv_key = handle.read()
 
-        signer = PythonRSASigner("", priv_key)
+        with open(ADB_PUB_KEY_PATH, "rb") as handle:
+            pub_key = handle.read()
+
+        signer = PythonRSASigner(pub_key, priv_key)
 
         # If no serial was specified or if the serial does not seem to be
         # a HOST:PORT definition, we use the USB transport.
@@ -84,53 +90,59 @@ class AndroidExtraction(MVTModule):
             except OSError as e:
                 if e.errno == 113 and self.serial:
                     log.critical("Unable to connect to the device %s: did you specify the correct IP addres?",
-                              self.serial)
+                                 self.serial)
                     sys.exit(-1)
             else:
                 break
 
     def _adb_disconnect(self):
-        """Close adb connection to the device.
-        """
+        """Close adb connection to the device."""
         self.device.close()
 
     def _adb_reconnect(self):
-        """Reconnect to device using adb.
-        """
+        """Reconnect to device using adb."""
         log.info("Reconnecting ...")
         self._adb_disconnect()
         self._adb_connect()
 
     def _adb_command(self, command):
         """Execute an adb shell command.
+
         :param command: Shell command to execute
         :returns: Output of command
+
         """
         return self.device.shell(command)
 
     def _adb_check_if_root(self):
         """Check if we have a `su` binary on the Android device.
+
+
         :returns: Boolean indicating whether a `su` binary is present or not
+
         """
         return bool(self._adb_command("command -v su"))
 
     def _adb_root_or_die(self):
-        """Check if we have a `su` binary, otherwise raise an Exception.
-        """
+        """Check if we have a `su` binary, otherwise raise an Exception."""
         if not self._adb_check_if_root():
             raise InsufficientPrivileges("This module is optionally available in case the device is already rooted. Do NOT root your own device!")
 
     def _adb_command_as_root(self, command):
         """Execute an adb shell command.
+
         :param command: Shell command to execute as root
         :returns: Output of command
+
         """
         return self._adb_command(f"su -c {command}")
-    
+
     def _adb_check_file_exists(self, file):
         """Verify that a file exists.
+
         :param file: Path of the file
         :returns: Boolean indicating whether the file exists or not
+
         """
 
         # TODO: Need to support checking files without root privileges as well.
@@ -144,9 +156,12 @@ class AndroidExtraction(MVTModule):
 
     def _adb_download(self, remote_path, local_path, progress_callback=None, retry_root=True):
         """Download a file form the device.
+
         :param remote_path: Path to download from the device
         :param local_path: Path to where to locally store the copy of the file
-        :param progress_callback: Callback for download progress bar
+        :param progress_callback: Callback for download progress bar (Default value = None)
+        :param retry_root: Default value = True)
+
         """
         try:
             self.device.pull(remote_path, local_path, progress_callback)
@@ -155,7 +170,7 @@ class AndroidExtraction(MVTModule):
                 self._adb_download_root(remote_path, local_path, progress_callback)
             else:
                 raise Exception(f"Unable to download file {remote_path}: {e}")
-    
+
     def _adb_download_root(self, remote_path, local_path, progress_callback=None):
         try:
             # Check if we have root, if not raise an Exception.
@@ -180,16 +195,18 @@ class AndroidExtraction(MVTModule):
 
             # Delete the copy on /sdcard/.
             self._adb_command(f"rm -rf {new_remote_path}")
-            
+
         except AdbCommandFailureException as e:
             raise Exception(f"Unable to download file {remote_path}: {e}")
 
     def _adb_process_file(self, remote_path, process_routine):
         """Download a local copy of a file which is only accessible as root.
         This is a wrapper around process_routine.
+
         :param remote_path: Path of the file on the device to process
         :param process_routine: Function to be called on the local copy of the
                                 downloaded file
+
         """
         # Connect to the device over adb.
         self._adb_connect()
@@ -223,6 +240,5 @@ class AndroidExtraction(MVTModule):
         self._adb_disconnect()
 
     def run(self):
-        """Run the main procedure.
-        """
+        """Run the main procedure."""
         raise NotImplementedError

mvt/android/modules/adb/chrome_history.py

@@ -16,6 +16,7 @@ log = logging.getLogger(__name__)
 
 CHROME_HISTORY_PATH = "data/data/com.android.chrome/app_chrome/Default/History"
 
+
 class ChromeHistory(AndroidExtraction):
     """This module extracts records from Android's Chrome browsing history."""
 
@@ -33,9 +34,19 @@ class ChromeHistory(AndroidExtraction):
             "data": f"{record['id']} - {record['url']} (visit ID: {record['visit_id']}, redirect source: {record['redirect_source']})"
         }
 
+    def check_indicators(self):
+        if not self.indicators:
+            return
+
+        for result in self.results:
+            if self.indicators.check_domain(result["url"]):
+                self.detected.append(result)
+
     def _parse_db(self, db_path):
         """Parse a Chrome History database file.
+
         :param db_path: Path to the History database to process.
+
         """
         conn = sqlite3.connect(db_path)
         cur = conn.cursor()

mvt/android/modules/adb/dumpsys_batterystats.py

@@ -10,6 +10,7 @@ from .base import AndroidExtraction
 
 log = logging.getLogger(__name__)
 
+
 class DumpsysBatterystats(AndroidExtraction):
     """This module extracts stats on battery consumption by processes."""
 
@@ -30,7 +31,7 @@ class DumpsysBatterystats(AndroidExtraction):
                 handle.write(stats)
 
             log.info("Records from dumpsys batterystats stored at %s",
-                      stats_path)
+                     stats_path)
 
         history = self._adb_command("dumpsys batterystats --history")
         if self.output_folder:

mvt/android/modules/adb/dumpsys_full.py

@@ -0,0 +1,36 @@
+# Mobile Verification Toolkit (MVT)
+# Copyright (c) 2021 The MVT Project Authors.
+# Use of this software is governed by the MVT License 1.1 that can be found at
+#   https://license.mvt.re/1.1/
+
+import logging
+import os
+
+from .base import AndroidExtraction
+
+log = logging.getLogger(__name__)
+
+
+class DumpsysFull(AndroidExtraction):
+    """This module extracts stats on battery consumption by processes."""
+
+    def __init__(self, file_path=None, base_folder=None, output_folder=None,
+                 serial=None, fast_mode=False, log=None, results=[]):
+        super().__init__(file_path=file_path, base_folder=base_folder,
+                         output_folder=output_folder, fast_mode=fast_mode,
+                         log=log, results=results)
+
+    def run(self):
+        self._adb_connect()
+
+        stats = self._adb_command("dumpsys")
+        if self.output_folder:
+            stats_path = os.path.join(self.output_folder,
+                                      "dumpsys.txt")
+            with open(stats_path, "w") as handle:
+                handle.write(stats)
+
+            log.info("Full dumpsys output stored at %s",
+                     stats_path)
+
+        self._adb_disconnect()

mvt/android/modules/adb/dumpsys_packages.py

@@ -10,8 +10,9 @@ from .base import AndroidExtraction
 
 log = logging.getLogger(__name__)
 
+
 class DumpsysPackages(AndroidExtraction):
-    """This module extracts stats on installed packages."""
+    """This module extracts details on installed packages."""
 
     def __init__(self, file_path=None, base_folder=None, output_folder=None,
                  serial=None, fast_mode=False, log=None, results=[]):
@@ -23,6 +24,7 @@ class DumpsysPackages(AndroidExtraction):
         self._adb_connect()
 
         output = self._adb_command("dumpsys package")
+
         if self.output_folder:
             packages_path = os.path.join(self.output_folder,
                                          "dumpsys_packages.txt")

mvt/android/modules/adb/dumpsys_procstats.py

@@ -10,6 +10,7 @@ from .base import AndroidExtraction
 
 log = logging.getLogger(__name__)
 
+
 class DumpsysProcstats(AndroidExtraction):
     """This module extracts stats on memory consumption by processes."""
 

mvt/android/modules/adb/dumpsys_receivers.py

@@ -0,0 +1,87 @@
+# Mobile Verification Toolkit (MVT)
+# Copyright (c) 2021 The MVT Project Authors.
+# Use of this software is governed by the MVT License 1.1 that can be found at
+#   https://license.mvt.re/1.1/
+
+import logging
+
+from .base import AndroidExtraction
+
+log = logging.getLogger(__name__)
+
+ACTION_NEW_OUTGOING_SMS = "android.provider.Telephony.NEW_OUTGOING_SMS"
+ACTION_SMS_RECEIVED = "android.provider.Telephony.SMS_RECEIVED"
+ACTION_DATA_SMS_RECEIVED = "android.intent.action.DATA_SMS_RECEIVED"
+ACTION_PHONE_STATE = "android.intent.action.PHONE_STATE"
+
+
+class DumpsysReceivers(AndroidExtraction):
+    """This module extracts details on receivers for risky activities."""
+
+    def __init__(self, file_path=None, base_folder=None, output_folder=None,
+                 serial=None, fast_mode=False, log=None, results=[]):
+        super().__init__(file_path=file_path, base_folder=base_folder,
+                         output_folder=output_folder, fast_mode=fast_mode,
+                         log=log, results=results)
+
+    def run(self):
+        self._adb_connect()
+
+        output = self._adb_command("dumpsys package")
+        if not output:
+            return
+
+        activity = None
+        for line in output.split("\n"):
+            # Find activity block markers.
+            if line.strip().startswith(ACTION_NEW_OUTGOING_SMS):
+                activity = ACTION_NEW_OUTGOING_SMS
+                continue
+            elif line.strip().startswith(ACTION_SMS_RECEIVED):
+                activity = ACTION_SMS_RECEIVED
+                continue
+            elif line.strip().startswith(ACTION_PHONE_STATE):
+                activity = ACTION_PHONE_STATE
+                continue
+            elif line.strip().startswith(ACTION_DATA_SMS_RECEIVED):
+                activity = ACTION_DATA_SMS_RECEIVED
+                continue
+
+            # If we are not in an activity block yet, skip.
+            if not activity:
+                continue
+
+            # If we are in a block but the line does not start with 8 spaces
+            # it means the block ended a new one started, so we reset and
+            # continue.
+            if not line.startswith(" " * 8):
+                activity = None
+                continue
+
+            # If we got this far, we are processing receivers for the
+            # activities we are interested in.
+            receiver = line.strip().split(" ")[1]
+            package_name = receiver.split("/")[0]
+            if package_name == "com.google.android.gms":
+                continue
+
+            if activity == ACTION_NEW_OUTGOING_SMS:
+                self.log.info("Found a receiver to intercept outgoing SMS messages: \"%s\"",
+                              receiver)
+            elif activity == ACTION_SMS_RECEIVED:
+                self.log.info("Found a receiver to intercept incoming SMS messages: \"%s\"",
+                              receiver)
+            elif activity == ACTION_DATA_SMS_RECEIVED:
+                self.log.info("Found a receiver to intercept incoming data SMS message: \"%s\"",
+                              receiver)
+            elif activity == ACTION_PHONE_STATE:
+                self.log.info("Found a receiver monitoring telephony state: \"%s\"",
+                              receiver)
+
+            self.results.append({
+                "activity": activity,
+                "package_name": package_name,
+                "receiver": receiver,
+            })
+
+        self._adb_disconnect()

mvt/android/modules/adb/files.py

@@ -0,0 +1,124 @@
+# Mobile Verification Toolkit (MVT)
+# Copyright (c) 2021 The MVT Project Authors.
+# Use of this software is governed by the MVT License 1.1 that can be found at
+#   https://license.mvt.re/1.1/
+
+import logging
+import os
+import stat
+import datetime
+
+from mvt.common.utils import check_for_links, convert_timestamp_to_iso
+
+from .base import AndroidExtraction
+
+log = logging.getLogger(__name__)
+
+
+class Files(AndroidExtraction):
+    """This module extracts the list of files on the device."""
+
+    def __init__(self, file_path=None, base_folder=None, output_folder=None,
+                 serial=None, fast_mode=False, log=None, results=[]):
+        super().__init__(file_path=file_path, base_folder=base_folder,
+                         output_folder=output_folder, fast_mode=fast_mode,
+                         log=log, results=results)
+        self.full_find = None
+
+    def find_path(self, file_path):
+        """Checks if Android system supports full find command output"""
+        # Check find command params on first run
+        # Run find command with correct args and parse results.
+
+        # Check that full file printf options are suppported on first run.
+        if self.full_find == None:
+            output = self._adb_command(f"find '/' -maxdepth 1 -printf '%T@ %m %s %u %g %p\n' 2> /dev/null")
+            if not (output or output.strip().splitlines()):
+                # Full  find command failed to generate output, fallback to basic file arguments
+                self.full_find = False
+            else:
+                self.full_find = True
+
+        found_files = []
+        if self.full_find == True:
+            # Run full file command and collect additonal file information.
+            output = self._adb_command(f"find '{file_path}' -printf '%T@ %m %s %u %g %p\n' 2> /dev/null")
+            for file_line in output.splitlines():
+                [unix_timestamp, mode, size, owner, group, full_path] = file_line.rstrip().split(" ", 5)
+                mod_time = convert_timestamp_to_iso(datetime.datetime.utcfromtimestamp(int(float(unix_timestamp))))
+                found_files.append({
+                    "path": full_path,
+                    "modified_time": mod_time,
+                    "mode": mode,
+                    "is_suid": (int(mode, 8) & stat.S_ISUID) == 2048,
+                    "is_sgid": (int(mode, 8) & stat.S_ISGID) == 1024,
+                    "size": size,
+                    "owner": owner,
+                    "group": group,
+                })
+        else:
+            # Run a basic listing of file paths.
+            output = self._adb_command(f"find '{file_path}' 2> /dev/null")
+            for file_line in output.splitlines():
+                found_files.append({
+                    "path": file_line.rstrip()
+                })
+
+        return found_files
+
+    def serialize(self, record):
+        if "modified_time" in record:
+            return {
+                "timestamp": record["modified_time"],
+                "module": self.__class__.__name__,
+                "event": "file_modified",
+                "data": record["path"],
+            }
+
+    def check_suspicious(self):
+        """Check for files with suspicious permissions"""
+        for result in sorted(self.results, key=lambda item: item["path"]):
+            if result.get("is_suid"):
+                self.log.warning("Found an SUID file in a non-standard directory \"%s\".",
+                                 result["path"])
+                self.detected.append(result)
+
+    def check_indicators(self):
+        """Check file list for known suspicious files or suspicious properties"""
+        self.check_suspicious()
+        if not self.indicators:
+            return
+
+        for result in self.results:
+            if self.indicators.check_filename(result["path"]):
+                self.log.warning("Found a known suspicous filename at path: \"%s\"", result["path"])
+                self.detected.append(result)
+
+            if self.indicators.check_file_path(result["path"]):
+                self.log.warning("Found a known suspicous file at path: \"%s\"", result["path"])
+                self.detected.append(result)
+
+    def run(self):
+        self._adb_connect()
+        found_file_paths = []
+
+        DATA_PATHS = ["/data/local/tmp/", "/sdcard/", "/tmp/"]
+        for path in DATA_PATHS:
+            file_info = self.find_path(path)
+            found_file_paths.extend(file_info)
+
+        # Store results
+        self.results.extend(found_file_paths)
+        self.log.info("Found %s files in primary Android data directories.", len(found_file_paths))
+
+        if self.fast_mode:
+            self.log.info("Flag --fast was enabled: skipping full file listing")
+        else:
+            self.log.info("Flag --fast was not enabled: processing full file listing. "
+                          "This may take a while...")
+            output = self.find_path("/")
+            if output and self.output_folder:
+                self.results.extend(output)
+                log.info("List of visible files stored in files.json")
+
+        self._adb_disconnect()

mvt/android/modules/adb/logcat.py

@@ -0,0 +1,48 @@
+# Mobile Verification Toolkit (MVT)
+# Copyright (c) 2021 The MVT Project Authors.
+# Use of this software is governed by the MVT License 1.1 that can be found at
+#   https://license.mvt.re/1.1/
+
+import logging
+import os
+
+from .base import AndroidExtraction
+
+log = logging.getLogger(__name__)
+
+
+class Logcat(AndroidExtraction):
+    """This module extracts details on installed packages."""
+
+    def __init__(self, file_path=None, base_folder=None, output_folder=None,
+                 serial=None, fast_mode=False, log=None, results=[]):
+        super().__init__(file_path=file_path, base_folder=base_folder,
+                         output_folder=output_folder, fast_mode=fast_mode,
+                         log=log, results=results)
+
+    def run(self):
+        self._adb_connect()
+
+        # Get the current logcat.
+        output = self._adb_command("logcat -d")
+        # Get the locat prior to last reboot.
+        last_output = self._adb_command("logcat -L")
+
+        if self.output_folder:
+            logcat_path = os.path.join(self.output_folder,
+                                       "logcat.txt")
+            with open(logcat_path, "w") as handle:
+                handle.write(output)
+
+            log.info("Current logcat logs stored at %s",
+                     logcat_path)
+
+            logcat_last_path = os.path.join(self.output_folder,
+                                            "logcat_last.txt")
+            with open(logcat_last_path, "w") as handle:
+                handle.write(last_output)
+
+            log.info("Logcat logs prior to last reboot stored at %s",
+                     logcat_last_path)
+
+        self._adb_disconnect()

mvt/android/modules/adb/packages.py

@@ -12,6 +12,7 @@ from .base import AndroidExtraction
 
 log = logging.getLogger(__name__)
 
+
 class Packages(AndroidExtraction):
     """This module extracts the list of installed packages."""
 
@@ -41,19 +42,54 @@ class Packages(AndroidExtraction):
         return records
 
     def check_indicators(self):
+        if not self.indicators:
+            return
+
         root_packages_path = os.path.join("..", "..", "data", "root_packages.txt")
         root_packages_string = pkg_resources.resource_string(__name__, root_packages_path)
         root_packages = root_packages_string.decode("utf-8").split("\n")
+        root_packages = [rp.strip() for rp in root_packages]
 
-        for root_package in root_packages:
-            root_package = root_package.strip()
-            if not root_package:
-                continue
-
-            if root_package in self.results:
+        for result in self.results:
+            if result["package_name"] in root_packages:
                 self.log.warning("Found an installed package related to rooting/jailbreaking: \"%s\"",
-                                 root_package)
-                self.detected.append(root_package)
+                                 result["package_name"])
+                self.detected.append(result)
+            if result["package_name"] in self.indicators.ioc_app_ids:
+                self.log.warning("Found a malicious package name: \"%s\"",
+                                 result["package_name"])
+                self.detected.append(result)
+                for file in result["files"]:
+                    if file["sha256"] in self.indicators.ioc_files_sha256:
+                        self.log.warning("Found a malicious APK: \"%s\" %s",
+                                         result["package_name"],
+                                         file["sha256"])
+                        self.detected.append(result)
+
+    def _get_files_for_package(self, package_name):
+        output = self._adb_command(f"pm path {package_name}")
+        output = output.strip().replace("package:", "")
+        if not output:
+            return []
+
+        package_files = []
+        for file_path in output.split("\n"):
+            file_path = file_path.strip()
+
+            md5 = self._adb_command(f"md5sum {file_path}").split(" ")[0]
+            sha1 = self._adb_command(f"sha1sum {file_path}").split(" ")[0]
+            sha256 = self._adb_command(f"sha256sum {file_path}").split(" ")[0]
+            sha512 = self._adb_command(f"sha512sum {file_path}").split(" ")[0]
+
+            package_files.append({
+                "path": file_path,
+                "md5": md5,
+                "sha1": sha1,
+                "sha256": sha256,
+                "sha512": sha512,
+            })
+
+        return package_files
 
     def run(self):
         self._adb_connect()
@@ -85,6 +121,8 @@ class Packages(AndroidExtraction):
             first_install = dumpsys[1].split("=")[1].strip()
             last_update = dumpsys[2].split("=")[1].strip()
 
+            package_files = self._get_files_for_package(package_name)
+
             self.results.append({
                 "package_name": package_name,
                 "file_name": file_name,
@@ -96,6 +134,7 @@ class Packages(AndroidExtraction):
                 "disabled": False,
                 "system": False,
                 "third_party": False,
+                "files": package_files,
             })
 
         cmds = [

mvt/android/modules/adb/processes.py

@@ -9,6 +9,7 @@ from .base import AndroidExtraction
 
 log = logging.getLogger(__name__)
 
+
 class Processes(AndroidExtraction):
     """This module extracts details on running processes."""
 

mvt/android/modules/adb/rootbinaries.py

@@ -12,6 +12,7 @@ from .base import AndroidExtraction
 
 log = logging.getLogger(__name__)
 
+
 class RootBinaries(AndroidExtraction):
     """This module extracts the list of installed packages."""
 

mvt/android/modules/adb/sms.py

@@ -15,12 +15,12 @@ log = logging.getLogger(__name__)
 
 SMS_BUGLE_PATH = "data/data/com.google.android.apps.messaging/databases/bugle_db"
 SMS_BUGLE_QUERY = """
-SELECT 
+SELECT
     ppl.normalized_destination AS number,
     p.timestamp AS timestamp,
-CASE WHEN m.sender_id IN 
+CASE WHEN m.sender_id IN
 (SELECT _id FROM participants WHERE contact_id=-1)
-THEN 2 ELSE 1 END incoming, p.text AS text 
+THEN 2 ELSE 1 END incoming, p.text AS text
 FROM messages m, conversations c, parts p,
         participants ppl, conversation_participants cp
 WHERE (m.conversation_id = c._id)
@@ -31,14 +31,15 @@ WHERE (m.conversation_id = c._id)
 
 SMS_MMSSMS_PATH = "data/data/com.android.providers.telephony/databases/mmssms.db"
 SMS_MMSMS_QUERY = """
-SELECT 
+SELECT
     address AS number,
     date_sent AS timestamp,
     type as incoming,
-    body AS text 
+    body AS text
 FROM sms;
 """
 
+
 class SMS(AndroidExtraction):
     """This module extracts all SMS messages containing links."""
 
@@ -62,7 +63,7 @@ class SMS(AndroidExtraction):
             return
 
         for message in self.results:
-            if not "text" in message:
+            if "text" not in message:
                 continue
 
             message_links = check_for_links(message["text"])
@@ -71,11 +72,13 @@ class SMS(AndroidExtraction):
 
     def _parse_db(self, db_path):
         """Parse an Android bugle_db SMS database file.
+
         :param db_path: Path to the Android SMS database file to process
+
         """
         conn = sqlite3.connect(db_path)
         cur = conn.cursor()
-        
+
         if (self.SMS_DB_TYPE == 1):
             cur.execute(SMS_BUGLE_QUERY)
         elif (self.SMS_DB_TYPE == 2):

mvt/android/modules/adb/whatsapp.py

@@ -16,6 +16,7 @@ log = logging.getLogger(__name__)
 
 WHATSAPP_PATH = "data/data/com.whatsapp/databases/msgstore.db"
 
+
 class Whatsapp(AndroidExtraction):
     """This module extracts all WhatsApp messages containing links."""
 
@@ -39,7 +40,7 @@ class Whatsapp(AndroidExtraction):
             return
 
         for message in self.results:
-            if not "data" in message:
+            if "data" not in message:
                 continue
 
             message_links = check_for_links(message["data"])
@@ -48,7 +49,9 @@ class Whatsapp(AndroidExtraction):
 
     def _parse_db(self, db_path):
         """Parse an Android msgstore.db WhatsApp database file.
+
         :param db_path: Path to the Android WhatsApp database file to process
+
         """
         conn = sqlite3.connect(db_path)
         cur = conn.cursor()

mvt/android/modules/backup/__init__.py

@@ -5,4 +5,4 @@
 
 from .sms import SMS
 
-BACKUP_MODULES = [SMS,]
+BACKUP_MODULES = [SMS]

mvt/android/modules/backup/sms.py

@@ -24,7 +24,7 @@ class SMS(MVTModule):
             return
 
         for message in self.results:
-            if not "body" in message:
+            if "body" not in message:
                 continue
 
             message_links = check_for_links(message["body"])

mvt/common/indicators.py

@@ -12,9 +12,12 @@ from .url import URL
 class IndicatorsFileBadFormat(Exception):
     pass
 
+
 class Indicators:
     """This class is used to parse indicators from a STIX2 file and provide
     functions to compare extracted artifacts to the indicators.
+
+
     """
 
     def __init__(self, log=None):
@@ -23,15 +26,35 @@ class Indicators:
         self.ioc_processes = []
         self.ioc_emails = []
         self.ioc_files = []
+        self.ioc_files_sha256 = []
+        self.ioc_app_ids = []
+        self.ios_profile_ids = []
         self.ioc_count = 0
+        self._check_env_variable()
 
     def _add_indicator(self, ioc, iocs_list):
         if ioc not in iocs_list:
             iocs_list.append(ioc)
             self.ioc_count += 1
 
+    def _check_env_variable(self):
+        """
+        Checks if a variable MVT_STIX2 contains path to STIX Files
+        """
+        if "MVT_STIX2" in os.environ:
+            paths = os.environ["MVT_STIX2"].split(":")
+            for path in paths:
+                if os.path.isfile(path):
+                    self.parse_stix2(path)
+                else:
+                    self.log.info("Invalid STIX2 path %s in MVT_STIX2 environment variable", path)
+
     def parse_stix2(self, file_path):
         """Extract indicators from a STIX2 file.
+
+        :param file_path: Path to the STIX2 file to parse
+        :type file_path: str
+
         """
         self.log.info("Parsing STIX2 indicators file at path %s",
                       file_path)
@@ -63,10 +86,29 @@ class Indicators:
             elif key == "file:name":
                 self._add_indicator(ioc=value,
                                     iocs_list=self.ioc_files)
+            elif key == "app:id":
+                self._add_indicator(ioc=value,
+                                    iocs_list=self.ioc_app_ids)
+            elif key == "configuration-profile:id":
+                self._add_indicator(ioc=value,
+                                    iocs_list=self.ios_profile_ids)
+            elif key == "file:hashes.sha256":
+                self._add_indicator(ioc=value,
+                                    iocs_list=self.ioc_files_sha256)
 
-    def check_domain(self, url):
+    def check_domain(self, url) -> bool:
+        """Check if a given URL matches any of the provided domain indicators.
+
+        :param url: URL to match against domain indicators
+        :type url: str
+        :returns: True if the URL matched an indicator, otherwise False
+        :rtype: bool
+
+        """
         # TODO: If the IOC domain contains a subdomain, it is not currently
         # being matched.
+        if not url:
+            return False
 
         try:
             # First we use the provided URL.
@@ -91,7 +133,7 @@ class Indicators:
             else:
                 # If it's not shortened, we just use the original URL object.
                 final_url = orig_url
-        except Exception as e:
+        except Exception:
             # If URL parsing failed, we just try to do a simple substring
             # match.
             for ioc in self.ioc_domains:
@@ -124,18 +166,35 @@ class Indicators:
 
                 return True
 
-    def check_domains(self, urls):
-        """Check the provided list of (suspicious) domains against a list of URLs.
-        :param urls: List of URLs to check
+        return False
+
+    def check_domains(self, urls) -> bool:
+        """Check a list of URLs against the provided list of domain indicators.
+
+        :param urls: List of URLs to check against domain indicators
+        :type urls: list
+        :returns: True if any URL matched an indicator, otherwise False
+        :rtype: bool
+
         """
+        if not urls:
+            return False
+
         for url in urls:
             if self.check_domain(url):
                 return True
 
-    def check_process(self, process):
+        return False
+
+    def check_process(self, process) -> bool:
         """Check the provided process name against the list of process
         indicators.
-        :param process: Process name to check
+
+        :param process: Process name to check against process indicators
+        :type process: str
+        :returns: True if process matched an indicator, otherwise False
+        :rtype: bool
+
         """
         if not process:
             return False
@@ -151,18 +210,35 @@ class Indicators:
                     self.log.warning("Found a truncated known suspicious process name \"%s\"", process)
                     return True
 
-    def check_processes(self, processes):
+        return False
+
+    def check_processes(self, processes) -> bool:
         """Check the provided list of processes against the list of
         process indicators.
-        :param processes: List of processes to check
+
+        :param processes: List of processes to check against process indicators
+        :type processes: list
+        :returns: True if process matched an indicator, otherwise False
+        :rtype: bool
+
         """
+        if not processes:
+            return False
+
         for process in processes:
             if self.check_process(process):
                 return True
 
-    def check_email(self, email):
+        return False
+
+    def check_email(self, email) -> bool:
         """Check the provided email against the list of email indicators.
-        :param email: Suspicious email to check
+
+        :param email: Email address to check against email indicators
+        :type email: str
+        :returns: True if email address matched an indicator, otherwise False
+        :rtype: bool
+
         """
         if not email:
             return False
@@ -171,14 +247,56 @@ class Indicators:
             self.log.warning("Found a known suspicious email address: \"%s\"", email)
             return True
 
-    def check_file(self, file_path):
+        return False
+
+    def check_filename(self, file_path) -> bool:
         """Check the provided file path against the list of file indicators.
-        :param file_path: Path or name of the file to check
+
+        :param file_path: File path or file name to check against file
+        indicators
+        :type file_path: str
+        :returns: True if the file path matched an indicator, otherwise False
+        :rtype: bool
+
         """
         if not file_path:
             return False
 
         file_name = os.path.basename(file_path)
         if file_name in self.ioc_files:
-            self.log.warning("Found a known suspicious file: \"%s\"", file_path)
             return True
+
+        return False
+
+    def check_file_path(self, file_path) -> bool:
+        """Check the provided file path against the list of file indicators.
+
+        :param file_path: File path or file name to check against file
+        indicators
+        :type file_path: str
+        :returns: True if the file path matched an indicator, otherwise False
+        :rtype: bool
+
+        """
+        if not file_path:
+            return False
+
+        for ioc_file in  self.ioc_files:
+            # Strip any trailing slash from indicator paths to match directories.
+            if file_path.startswith(ioc_file.rstrip("/")):
+                return True
+        return False
+
+    def check_profile(self, profile_uuid) -> bool:
+        """Check the provided configuration profile UUID against the list of indicators.
+
+        :param profile_uuid: Profile UUID to check against configuration profile indicators
+        :type profile_uuid: str
+        :returns: True if the UUID in indicator list, otherwise False
+        :rtype: bool
+
+        """
+        if profile_uuid in self.ios_profile_ids:
+            return True
+
+        return False

mvt/common/logo.py

@@ -0,0 +1,25 @@
+# Mobile Verification Toolkit (MVT)
+# Copyright (c) 2021 The MVT Project Authors.
+# Use of this software is governed by the MVT License 1.1 that can be found at
+#   https://license.mvt.re/1.1/
+
+from rich import print
+
+from .version import MVT_VERSION, check_for_updates
+
+
+def logo():
+    print("\n")
+    print("\t[bold]MVT[/bold] - Mobile Verification Toolkit")
+    print("\t\thttps://mvt.re")
+    print(f"\t\tVersion: {MVT_VERSION}")
+
+    try:
+        latest_version = check_for_updates()
+    except Exception:
+        pass
+    else:
+        if latest_version:
+            print(f"\t\t[bold]Version {latest_version} is available! Upgrade mvt![/bold]")
+
+    print("\n")

mvt/common/module.py

@@ -10,18 +10,19 @@ import re
 
 import simplejson as json
 
-from .indicators import Indicators
-
 
 class DatabaseNotFoundError(Exception):
     pass
 
+
 class DatabaseCorruptedError(Exception):
     pass
 
+
 class InsufficientPrivileges(Exception):
     pass
 
+
 class MVTModule(object):
     """This class provides a base for all extraction modules."""
 
@@ -31,12 +32,18 @@ class MVTModule(object):
     def __init__(self, file_path=None, base_folder=None, output_folder=None,
                  fast_mode=False, log=None, results=[]):
         """Initialize module.
-        :param file_path: Path to the module's database file, if there is any.
+
+        :param file_path: Path to the module's database file, if there is any
+        :type file_path: str
         :param base_folder: Path to the base folder (backup or filesystem dump)
+        :type file_path: str
         :param output_folder: Folder where results will be stored
+        :type output_folder: str
         :param fast_mode: Flag to enable or disable slow modules
+        :type fast_mode: bool
         :param log: Handle to logger
         :param results: Provided list of results entries
+        :type results: list
         """
         self.file_path = file_path
         self.base_folder = base_folder
@@ -59,23 +66,23 @@ class MVTModule(object):
             return cls(results=results, log=log)
 
     def get_slug(self):
+        """Use the module's class name to retrieve a slug"""
         if self.slug:
             return self.slug
 
         sub = re.sub("(.)([A-Z][a-z]+)", r"\1_\2", self.__class__.__name__)
         return re.sub("([a-z0-9])([A-Z])", r"\1_\2", sub).lower()
 
-    def load_indicators(self, file_path):
-        self.indicators = Indicators(file_path, self.log)
-
     def check_indicators(self):
         """Check the results of this module against a provided list of
-        indicators."""
+        indicators.
+
+
+        """
         raise NotImplementedError
 
     def save_to_json(self):
-        """Save the collected results to a json file.
-        """
+        """Save the collected results to a json file."""
         if not self.output_folder:
             return
 
@@ -100,9 +107,20 @@ class MVTModule(object):
     def serialize(self, record):
         raise NotImplementedError
 
-    def to_timeline(self):
-        """Convert results into a timeline.
+    @staticmethod
+    def _deduplicate_timeline(timeline):
+        """Serialize entry as JSON to deduplicate repeated entries
+
+        :param timeline: List of entries from timeline to deduplicate
+
         """
+        timeline_set = set()
+        for record in timeline:
+            timeline_set.add(json.dumps(record, sort_keys=True))
+        return [json.loads(record) for record in timeline_set]
+
+    def to_timeline(self):
+        """Convert results into a timeline."""
         for result in self.results:
             record = self.serialize(result)
             if record:
@@ -120,19 +138,11 @@ class MVTModule(object):
                     self.timeline_detected.append(record)
 
         # De-duplicate timeline entries.
-        self.timeline = self.timeline_deduplicate(self.timeline)
-        self.timeline_detected = self.timeline_deduplicate(self.timeline_detected)
-
-    def timeline_deduplicate(self, timeline):
-        """Serialize entry as JSON to deduplicate repeated entries"""
-        timeline_set = set()
-        for record in timeline:
-            timeline_set.add(json.dumps(record, sort_keys=True))
-        return [json.loads(record) for record in timeline_set]
+        self.timeline = self._deduplicate_timeline(self.timeline)
+        self.timeline_detected = self._deduplicate_timeline(self.timeline_detected)
 
     def run(self):
-        """Run the main module procedure.
-        """
+        """Run the main module procedure."""
         raise NotImplementedError
 
 
@@ -150,7 +160,7 @@ def run_module(module):
         module.log.info("There might be no data to extract by module %s: %s",
                         module.__class__.__name__, e)
     except DatabaseCorruptedError as e:
-        module.log.error("The %s module database seems to be corrupted and recovery failed: %s",
+        module.log.error("The %s module database seems to be corrupted: %s",
                          module.__class__.__name__, e)
     except Exception as e:
         module.log.exception("Error in running extraction from module %s: %s",
@@ -177,8 +187,10 @@ def run_module(module):
 
 def save_timeline(timeline, timeline_path):
     """Save the timeline in a csv file.
-    :param timeline: List of records to order and store.
-    :param timeline_path: Path to the csv file to store the timeline to.
+
+    :param timeline: List of records to order and store
+    :param timeline_path: Path to the csv file to store the timeline to
+
     """
     with io.open(timeline_path, "a+", encoding="utf-8") as handle:
         csvoutput = csv.writer(handle, delimiter=",", quotechar="\"")