Bumped version

This way when a detection is logged, the user can know which STIX2
file was matched by the module

.github/workflows/python-package.yml

@@ -16,7 +16,8 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: [3.7, 3.8, 3.9]
+        # python-version: [3.7, 3.8, 3.9]
+        python-version: [3.8, 3.9]
 
     steps:
     - uses: actions/checkout@v2
@@ -27,8 +28,9 @@ jobs:
     - name: Install dependencies
       run: |
         python -m pip install --upgrade pip
-        python -m pip install flake8 pytest safety
+        python -m pip install flake8 pytest safety stix2
         if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
+        python -m pip install .
     - name: Lint with flake8
       run: |
         # stop the build if there are Python syntax errors or undefined names
@@ -37,7 +39,5 @@ jobs:
         flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
     - name: Safety checks
       run: safety check
-
-    # - name: Test with pytest
-    #   run: |
-    #     pytest
+    - name: Test with pytest
+      run: pytest

Dockerfile

@@ -38,12 +38,15 @@ RUN apt update \
 # Build libimobiledevice
 # ----------------------
 RUN git clone https://github.com/libimobiledevice/libplist \
+  && git clone https://github.com/libimobiledevice/libimobiledevice-glue \
   && git clone https://github.com/libimobiledevice/libusbmuxd \
   && git clone https://github.com/libimobiledevice/libimobiledevice \
   && git clone https://github.com/libimobiledevice/usbmuxd \
 
   && cd libplist && ./autogen.sh && make && make install && ldconfig \
 
+  && cd ../libimobiledevice-glue && PKG_CONFIG_PATH=/usr/local/lib/pkgconfig ./autogen.sh --prefix=/usr && make && make install && ldconfig \
+
   && cd ../libusbmuxd && PKG_CONFIG_PATH=/usr/local/lib/pkgconfig ./autogen.sh && make && make install && ldconfig \
 
   && cd ../libimobiledevice && PKG_CONFIG_PATH=/usr/local/lib/pkgconfig ./autogen.sh --enable-debug && make && make install && ldconfig \
@@ -51,7 +54,7 @@ RUN git clone https://github.com/libimobiledevice/libplist \
   && cd ../usbmuxd && PKG_CONFIG_PATH=/usr/local/lib/pkgconfig ./autogen.sh --prefix=/usr --sysconfdir=/etc --localstatedir=/var --runstatedir=/run && make && make install \
 
   # Clean up.
-  && cd .. && rm -rf libplist libusbmuxd libimobiledevice usbmuxd
+  && cd .. && rm -rf libplist libimobiledevice-glue libusbmuxd libimobiledevice usbmuxd
 
 # Installing MVT
 # --------------

README.md

@@ -15,15 +15,15 @@ It has been developed and released by the [Amnesty International Security Lab](h
 
 ## Installation
 
-MVT can be installed from sources or from [PyPi](https://pypi.org/project/mvt/) (you will need some dependencies, check the [documentation](https://docs.mvt.re/en/latest/install.html)):
+MVT can be installed from sources or from [PyPi](https://pypi.org/project/mvt/) (you will need some dependencies, check the [documentation](https://docs.mvt.re/en/latest/install/)):
 
 ```
 pip3 install mvt
 ```
 
-Alternatively, you can decide to run MVT and all relevant tools through a [Docker container](https://docs.mvt.re/en/latest/docker.html).
+Alternatively, you can decide to run MVT and all relevant tools through a [Docker container](https://docs.mvt.re/en/latest/docker/).
 
-**Please note:** MVT is best run on Linux or Mac systems. [It does not currently support running natively on Windows.](https://docs.mvt.re/en/latest/install.html#mvt-on-windows)
+**Please note:** MVT is best run on Linux or Mac systems. [It does not currently support running natively on Windows.](https://docs.mvt.re/en/latest/install/#mvt-on-windows)
 
 ## Usage
 
@@ -31,4 +31,4 @@ MVT provides two commands `mvt-ios` and `mvt-android`. [Check out the documentat
 
 ## License
 
-The purpose of MVT is to facilitate the ***consensual forensic analysis*** of devices of those who might be targets of sophisticated mobile spyware attacks, especially members of civil society and marginalized communities. We do not want MVT to enable privacy violations of non-consenting individuals.  In order to achieve this, MVT is released under its own license. [Read more here.](https://docs.mvt.re/en/latest/license.html)
+The purpose of MVT is to facilitate the ***consensual forensic analysis*** of devices of those who might be targets of sophisticated mobile spyware attacks, especially members of civil society and marginalized communities. We do not want MVT to enable privacy violations of non-consenting individuals.  In order to achieve this, MVT is released under its own license. [Read more here.](https://docs.mvt.re/en/latest/license/)

docs/android/backup.md

@@ -22,7 +22,7 @@ adb backup -all
 
 ## Unpack the backup
 
-In order to reliable unpack th [Android Backup Extractor (ABE)](https://github.com/nelenkov/android-backup-extractor) to convert it to a readable file format. Make sure that java is installed on your system and use the following command:
+In order to unpack the backup, use [Android Backup Extractor (ABE)](https://github.com/nelenkov/android-backup-extractor) to convert it to a readable file format. Make sure that java is installed on your system and use the following command:
 
 ```bash
 java -jar ~/path/to/abe.jar unpack backup.ab backup.tar
@@ -31,6 +31,8 @@ tar xvf backup.tar
 
 If the backup is encrypted, ABE will prompt you to enter the password.
 
+Alternatively, [ab-decrypt](https://github.com/joernheissler/ab-decrypt) can be used for that purpose.
+
 ## Check the backup
 
 You can then extract SMSs containing links with MVT:

docs/iocs.md

@@ -28,9 +28,19 @@ The `--iocs` option can be invoked multiple times to let MVT import multiple STI
 mvt-ios check-backup --iocs ~/iocs/malware1.stix --iocs ~/iocs/malware2.stix2 /path/to/backup
 ```
 
+It is also possible to load STIX2 files automatically from the environment variable `MVT_STIX2`:
+
+```bash
+export MVT_STIX2="/home/user/IOC1.stix2:/home/user/IOC2.stix2"
+```
+
 ## Known repositories of STIX2 IOCs
 
 - The [Amnesty International investigations repository](https://github.com/AmnestyTech/investigations) contains STIX-formatted IOCs for:
     - [Pegasus](https://en.wikipedia.org/wiki/Pegasus_(spyware)) ([STIX2](https://raw.githubusercontent.com/AmnestyTech/investigations/master/2021-07-18_nso/pegasus.stix2))
+    - [Predator from Cytrox](https://citizenlab.ca/2021/12/pegasus-vs-predator-dissidents-doubly-infected-iphone-reveals-cytrox-mercenary-spyware/) ([STIX2](https://raw.githubusercontent.com/AmnestyTech/investigations/master/2021-12-16_cytrox/cytrox.stix2))
+- [This repository](https://github.com/Te-k/stalkerware-indicators) contains IOCs for Android stalkerware including [a STIX MVT-compatible file](https://raw.githubusercontent.com/Te-k/stalkerware-indicators/master/stalkerware.stix2).
+
+You can automaticallly download the latest public indicator files with the command `mvt-ios download-iocs` or `mvt-android download-iocs`. These commands download the list of indicators listed [here](https://github.com/mvt-project/mvt/blob/main/public_indicators.json) and store them in the [appdir](https://pypi.org/project/appdirs/) folder. They are then loaded automatically by mvt.
 
 Please [open an issue](https://github.com/mvt-project/mvt/issues/) to suggest new sources of STIX-formatted IOCs.

docs/requirements.txt

@@ -1,4 +1,4 @@
-mkdocs==1.2.1
+mkdocs==1.2.3
 mkdocs-autorefs
 mkdocs-material
 mkdocs-material-extensions

mvt/android/cli.py

@@ -9,8 +9,10 @@ import os
 import click
 from rich.logging import RichHandler
 
-from mvt.common.help import *
-from mvt.common.indicators import Indicators, IndicatorsFileBadFormat
+from mvt.common.help import (HELP_MSG_FAST, HELP_MSG_IOC,
+                             HELP_MSG_LIST_MODULES, HELP_MSG_MODULE,
+                             HELP_MSG_OUTPUT, HELP_MSG_SERIAL)
+from mvt.common.indicators import Indicators, download_indicators_files
 from mvt.common.logo import logo
 from mvt.common.module import run_module, save_timeline
 
@@ -26,6 +28,7 @@ logging.basicConfig(level="INFO", format=LOG_FORMAT, handlers=[
     RichHandler(show_path=False, log_time_format="%X")])
 log = logging.getLogger(__name__)
 
+
 #==============================================================================
 # Main
 #==============================================================================
@@ -104,10 +107,11 @@ def download_apks(ctx, all_apks, virustotal, koodous, all_checks, output, from_f
               default=[], help=HELP_MSG_IOC)
 @click.option("--output", "-o", type=click.Path(exists=False),
               help=HELP_MSG_OUTPUT)
+@click.option("--fast", "-f", is_flag=True, help=HELP_MSG_FAST)
 @click.option("--list-modules", "-l", is_flag=True, help=HELP_MSG_LIST_MODULES)
 @click.option("--module", "-m", help=HELP_MSG_MODULE)
 @click.pass_context
-def check_adb(ctx, iocs, output, list_modules, module, serial):
+def check_adb(ctx, iocs, output, fast, list_modules, module, serial):
     if list_modules:
         log.info("Following is the list of available check-adb modules:")
         for adb_module in ADB_MODULES:
@@ -125,13 +129,7 @@ def check_adb(ctx, iocs, output, list_modules, module, serial):
             ctx.exit(1)
 
     indicators = Indicators(log=log)
-    for ioc_path in iocs:
-        try:
-            indicators.parse_stix2(ioc_path)
-        except IndicatorsFileBadFormat as e:
-            log.critical(e)
-            ctx.exit(1)
-    log.info("Loaded a total of %d indicators", indicators.ioc_count)
+    indicators.load_indicators_files(iocs)
 
     timeline = []
     timeline_detected = []
@@ -139,14 +137,14 @@ def check_adb(ctx, iocs, output, list_modules, module, serial):
         if module and adb_module.__name__ != module:
             continue
 
-        m = adb_module(output_folder=output, log=logging.getLogger(adb_module.__module__))
+        m = adb_module(output_folder=output, fast_mode=fast,
+                       log=logging.getLogger(adb_module.__module__))
+        if indicators.total_ioc_count:
+            m.indicators = indicators
+            m.indicators.log = m.log
         if serial:
             m.serial = serial
 
-        if iocs:
-            indicators.log = m.log
-            m.indicators = indicators
-
         run_module(m)
         timeline.extend(m.timeline)
         timeline_detected.extend(m.timeline_detected)
@@ -179,31 +177,31 @@ def check_backup(ctx, iocs, output, backup_path, serial):
             ctx.exit(1)
 
     indicators = Indicators(log=log)
-    for ioc_path in iocs:
-        try:
-            indicators.parse_stix2(ioc_path)
-        except IndicatorsFileBadFormat as e:
-            log.critical(e)
-            ctx.exit(1)
-    log.info("Loaded a total of %d indicators", indicators.ioc_count)
+    indicators.load_indicators_files(iocs)
 
     if os.path.isfile(backup_path):
         log.critical("The path you specified is a not a folder!")
 
         if os.path.basename(backup_path) == "backup.ab":
-            log.info("You can use ABE (https://github.com/nelenkov/android-backup-extractor) " \
+            log.info("You can use ABE (https://github.com/nelenkov/android-backup-extractor) "
                      "to extract 'backup.ab' files!")
         ctx.exit(1)
 
     for module in BACKUP_MODULES:
         m = module(base_folder=backup_path, output_folder=output,
                    log=logging.getLogger(module.__module__))
-
+        if indicators.total_ioc_count:
+            m.indicators = indicators
+            m.indicators.log = m.log
         if serial:
             m.serial = serial
 
-        if iocs:
-            indicators.log = m.log
-            m.indicators = indicators
-
         run_module(m)
+
+
+#==============================================================================
+# Command: download-iocs
+#==============================================================================
+@cli.command("download-iocs", help="Download public STIX2 indicators")
+def download_indicators():
+    download_indicators_files(log)

mvt/android/download_apks.py

@@ -7,7 +7,6 @@ import json
 import logging
 import os
 
-import pkg_resources
 from tqdm import tqdm
 
 from mvt.common.module import InsufficientPrivileges
@@ -17,6 +16,7 @@ from .modules.adb.packages import Packages
 
 log = logging.getLogger(__name__)
 
+
 # TODO: Would be better to replace tqdm with rich.progress to reduce
 #       the number of dependencies. Need to investigate whether
 #       it's possible to have a similar callback system.
@@ -138,7 +138,7 @@ class DownloadAPKs(AndroidExtraction):
                     packages_selection.append(package)
 
             log.info("Selected only %d packages which are not marked as system",
-                len(packages_selection))
+                     len(packages_selection))
 
         if len(packages_selection) == 0:
             log.info("No packages were selected for download")

mvt/android/lookups/koodous.py

@@ -13,6 +13,7 @@ from rich.text import Text
 
 log = logging.getLogger(__name__)
 
+
 def koodous_lookup(packages):
     log.info("Looking up all extracted files on Koodous (www.koodous.com)")
     log.info("This might take a while...")

mvt/android/lookups/virustotal.py

@@ -13,6 +13,7 @@ from rich.text import Text
 
 log = logging.getLogger(__name__)
 
+
 def get_virustotal_report(hashes):
     apikey = "233f22e200ca5822bd91103043ccac138b910db79f29af5616a9afe8b6f215ad"
     url = f"https://www.virustotal.com/partners/sysinternals/file-reports?apikey={apikey}"
@@ -36,6 +37,7 @@ def get_virustotal_report(hashes):
         log.error("Unexpected response from VirusTotal: %s", res.status_code)
         return None
 
+
 def virustotal_lookup(packages):
     log.info("Looking up all extracted files on VirusTotal (www.virustotal.com)")
 
@@ -48,6 +50,7 @@ def virustotal_lookup(packages):
     total_unique_hashes = len(unique_hashes)
 
     detections = {}
+
     def virustotal_query(batch):
         report = get_virustotal_report(batch)
         if not report:

mvt/android/modules/adb/__init__.py

@@ -4,6 +4,7 @@
 #   https://license.mvt.re/1.1/
 
 from .chrome_history import ChromeHistory
+from .dumpsys_accessibility import DumpsysAccessibility
 from .dumpsys_batterystats import DumpsysBatterystats
 from .dumpsys_full import DumpsysFull
 from .dumpsys_packages import DumpsysPackages
@@ -18,6 +19,6 @@ from .sms import SMS
 from .whatsapp import Whatsapp
 
 ADB_MODULES = [ChromeHistory, SMS, Whatsapp, Processes,
-               DumpsysBatterystats, DumpsysProcstats,
+               DumpsysAccessibility, DumpsysBatterystats, DumpsysProcstats,
                DumpsysPackages, DumpsysReceivers, DumpsysFull,
                Packages, RootBinaries, Logcat, Files]

mvt/android/modules/adb/base.py

@@ -25,6 +25,7 @@ log = logging.getLogger(__name__)
 ADB_KEY_PATH = os.path.expanduser("~/.android/adbkey")
 ADB_PUB_KEY_PATH = os.path.expanduser("~/.android/adbkey.pub")
 
+
 class AndroidExtraction(MVTModule):
     """This class provides a base for all Android extraction modules."""
 
@@ -41,7 +42,7 @@ class AndroidExtraction(MVTModule):
     def _adb_check_keys():
         """Make sure Android adb keys exist."""
         if not os.path.isdir(os.path.dirname(ADB_KEY_PATH)):
-            os.path.makedirs(os.path.dirname(ADB_KEY_PATH))
+            os.makedirs(os.path.dirname(ADB_KEY_PATH))
 
         if not os.path.exists(ADB_KEY_PATH):
             keygen(ADB_KEY_PATH)
@@ -56,7 +57,10 @@ class AndroidExtraction(MVTModule):
         with open(ADB_KEY_PATH, "rb") as handle:
             priv_key = handle.read()
 
-        signer = PythonRSASigner("", priv_key)
+        with open(ADB_PUB_KEY_PATH, "rb") as handle:
+            pub_key = handle.read()
+
+        signer = PythonRSASigner(pub_key, priv_key)
 
         # If no serial was specified or if the serial does not seem to be
         # a HOST:PORT definition, we use the USB transport.
@@ -86,7 +90,7 @@ class AndroidExtraction(MVTModule):
             except OSError as e:
                 if e.errno == 113 and self.serial:
                     log.critical("Unable to connect to the device %s: did you specify the correct IP addres?",
-                              self.serial)
+                                 self.serial)
                     sys.exit(-1)
             else:
                 break
@@ -108,7 +112,7 @@ class AndroidExtraction(MVTModule):
         :returns: Output of command
 
         """
-        return self.device.shell(command)
+        return self.device.shell(command, read_timeout_s=200.0)
 
     def _adb_check_if_root(self):
         """Check if we have a `su` binary on the Android device.

mvt/android/modules/adb/chrome_history.py

@@ -16,6 +16,7 @@ log = logging.getLogger(__name__)
 
 CHROME_HISTORY_PATH = "data/data/com.android.chrome/app_chrome/Default/History"
 
+
 class ChromeHistory(AndroidExtraction):
     """This module extracts records from Android's Chrome browsing history."""
 

mvt/android/modules/adb/dumpsys_accessibility.py

@@ -0,0 +1,53 @@
+# Mobile Verification Toolkit (MVT)
+# Copyright (c) 2021 The MVT Project Authors.
+# Use of this software is governed by the MVT License 1.1 that can be found at
+#   https://license.mvt.re/1.1/
+
+import io
+import logging
+import os
+
+from .base import AndroidExtraction
+
+log = logging.getLogger(__name__)
+
+
+class DumpsysAccessibility(AndroidExtraction):
+    """This module extracts stats on accessibility."""
+
+    def __init__(self, file_path=None, base_folder=None, output_folder=None,
+                 serial=None, fast_mode=False, log=None, results=[]):
+        super().__init__(file_path=file_path, base_folder=base_folder,
+                         output_folder=output_folder, fast_mode=fast_mode,
+                         log=log, results=results)
+
+    def run(self):
+        self._adb_connect()
+
+        stats = self._adb_command("dumpsys accessibility")
+
+        in_services = False
+        for line in stats.split("\n"):
+            if line.strip().startswith("installed services:"):
+                in_services = True
+                continue
+
+            if not in_services:
+                continue
+
+            if line.strip() == "}":
+                break
+
+            service = line.split(":")[1].strip()
+            log.info("Found installed accessibility service \"%s\"", service)
+
+        if self.output_folder:
+            acc_path = os.path.join(self.output_folder,
+                                      "dumpsys_accessibility.txt")
+            with io.open(acc_path, "w", encoding="utf-8") as handle:
+                handle.write(stats)
+
+            log.info("Records from dumpsys accessibility stored at %s",
+                     acc_path)
+
+        self._adb_disconnect()

mvt/android/modules/adb/dumpsys_batterystats.py

@@ -10,6 +10,7 @@ from .base import AndroidExtraction
 
 log = logging.getLogger(__name__)
 
+
 class DumpsysBatterystats(AndroidExtraction):
     """This module extracts stats on battery consumption by processes."""
 
@@ -30,7 +31,7 @@ class DumpsysBatterystats(AndroidExtraction):
                 handle.write(stats)
 
             log.info("Records from dumpsys batterystats stored at %s",
-                      stats_path)
+                     stats_path)
 
         history = self._adb_command("dumpsys batterystats --history")
         if self.output_folder:

mvt/android/modules/adb/dumpsys_full.py

@@ -10,6 +10,7 @@ from .base import AndroidExtraction
 
 log = logging.getLogger(__name__)
 
+
 class DumpsysFull(AndroidExtraction):
     """This module extracts stats on battery consumption by processes."""
 
@@ -30,6 +31,6 @@ class DumpsysFull(AndroidExtraction):
                 handle.write(stats)
 
             log.info("Full dumpsys output stored at %s",
-                      stats_path)
+                     stats_path)
 
         self._adb_disconnect()

mvt/android/modules/adb/dumpsys_procstats.py

@@ -10,6 +10,7 @@ from .base import AndroidExtraction
 
 log = logging.getLogger(__name__)
 
+
 class DumpsysProcstats(AndroidExtraction):
     """This module extracts stats on memory consumption by processes."""
 

mvt/android/modules/adb/dumpsys_receivers.py

@@ -4,7 +4,6 @@
 #   https://license.mvt.re/1.1/
 
 import logging
-import os
 
 from .base import AndroidExtraction
 
@@ -15,6 +14,7 @@ ACTION_SMS_RECEIVED = "android.provider.Telephony.SMS_RECEIVED"
 ACTION_DATA_SMS_RECEIVED = "android.intent.action.DATA_SMS_RECEIVED"
 ACTION_PHONE_STATE = "android.intent.action.PHONE_STATE"
 
+
 class DumpsysReceivers(AndroidExtraction):
     """This module extracts details on receivers for risky activities."""
 
@@ -67,16 +67,16 @@ class DumpsysReceivers(AndroidExtraction):
 
             if activity == ACTION_NEW_OUTGOING_SMS:
                 self.log.info("Found a receiver to intercept outgoing SMS messages: \"%s\"",
-                    receiver)
+                              receiver)
             elif activity == ACTION_SMS_RECEIVED:
                 self.log.info("Found a receiver to intercept incoming SMS messages: \"%s\"",
-                    receiver)
+                              receiver)
             elif activity == ACTION_DATA_SMS_RECEIVED:
                 self.log.info("Found a receiver to intercept incoming data SMS message: \"%s\"",
-                    receiver)
+                              receiver)
             elif activity == ACTION_PHONE_STATE:
                 self.log.info("Found a receiver monitoring telephony state: \"%s\"",
-                    receiver)
+                              receiver)
 
             self.results.append({
                 "activity": activity,

mvt/android/modules/adb/files.py

@@ -3,31 +3,117 @@
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 
+import datetime
 import logging
-import os
+import stat
+
+from mvt.common.utils import convert_timestamp_to_iso
 
 from .base import AndroidExtraction
 
 log = logging.getLogger(__name__)
 
+
 class Files(AndroidExtraction):
-    """This module extracts the list of installed packages."""
+    """This module extracts the list of files on the device."""
 
     def __init__(self, file_path=None, base_folder=None, output_folder=None,
                  serial=None, fast_mode=False, log=None, results=[]):
         super().__init__(file_path=file_path, base_folder=base_folder,
                          output_folder=output_folder, fast_mode=fast_mode,
                          log=log, results=results)
+        self.full_find = None
+
+    def find_path(self, file_path):
+        """Checks if Android system supports full find command output"""
+        # Check find command params on first run
+        # Run find command with correct args and parse results.
+
+        # Check that full file printf options are suppported on first run.
+        if self.full_find is None:
+            output = self._adb_command("find '/' -maxdepth 1 -printf '%T@ %m %s %u %g %p\n' 2> /dev/null")
+            if not (output or output.strip().splitlines()):
+                # Full  find command failed to generate output, fallback to basic file arguments
+                self.full_find = False
+            else:
+                self.full_find = True
+
+        found_files = []
+        if self.full_find is True:
+            # Run full file command and collect additonal file information.
+            output = self._adb_command(f"find '{file_path}' -printf '%T@ %m %s %u %g %p\n' 2> /dev/null")
+            for file_line in output.splitlines():
+                [unix_timestamp, mode, size, owner, group, full_path] = file_line.rstrip().split(" ", 5)
+                mod_time = convert_timestamp_to_iso(datetime.datetime.utcfromtimestamp(int(float(unix_timestamp))))
+                found_files.append({
+                    "path": full_path,
+                    "modified_time": mod_time,
+                    "mode": mode,
+                    "is_suid": (int(mode, 8) & stat.S_ISUID) == 2048,
+                    "is_sgid": (int(mode, 8) & stat.S_ISGID) == 1024,
+                    "size": size,
+                    "owner": owner,
+                    "group": group,
+                })
+        else:
+            # Run a basic listing of file paths.
+            output = self._adb_command(f"find '{file_path}' 2> /dev/null")
+            for file_line in output.splitlines():
+                found_files.append({
+                    "path": file_line.rstrip()
+                })
+
+        return found_files
+
+    def serialize(self, record):
+        if "modified_time" in record:
+            return {
+                "timestamp": record["modified_time"],
+                "module": self.__class__.__name__,
+                "event": "file_modified",
+                "data": record["path"],
+            }
+
+    def check_suspicious(self):
+        """Check for files with suspicious permissions"""
+        for result in sorted(self.results, key=lambda item: item["path"]):
+            if result.get("is_suid"):
+                self.log.warning("Found an SUID file in a non-standard directory \"%s\".",
+                                 result["path"])
+                self.detected.append(result)
+
+    def check_indicators(self):
+        """Check file list for known suspicious files or suspicious properties"""
+        self.check_suspicious()
+        if not self.indicators:
+            return
+
+        for result in self.results:
+            if self.indicators.check_file_path(result["path"]):
+                self.log.warning("Found a known suspicous file at path: \"%s\"", result["path"])
+                self.detected.append(result)
 
     def run(self):
         self._adb_connect()
+        found_file_paths = []
 
-        output = self._adb_command("find / -type f 2> /dev/null")
-        if output and self.output_folder:
-            files_txt_path = os.path.join(self.output_folder, "files.txt")
-            with open(files_txt_path, "w") as handle:
-                handle.write(output)
+        DATA_PATHS = ["/data/local/tmp/", "/sdcard/", "/tmp/"]
+        for path in DATA_PATHS:
+            file_info = self.find_path(path)
+            found_file_paths.extend(file_info)
 
-            log.info("List of visible files stored at %s", files_txt_path)
+        # Store results
+        self.results.extend(found_file_paths)
+        self.log.info("Found %s files in primary Android data directories.", len(found_file_paths))
+
+        if self.fast_mode:
+            self.log.info("Flag --fast was enabled: skipping full file listing")
+        else:
+            self.log.info("Flag --fast was not enabled: processing full file listing. "
+                          "This may take a while...")
+            output = self.find_path("/")
+            if output and self.output_folder:
+                self.results.extend(output)
+                log.info("List of visible files stored in files.json")
 
         self._adb_disconnect()

mvt/android/modules/adb/packages.py

@@ -12,6 +12,7 @@ from .base import AndroidExtraction
 
 log = logging.getLogger(__name__)
 
+
 class Packages(AndroidExtraction):
     """This module extracts the list of installed packages."""
 
@@ -41,16 +42,18 @@ class Packages(AndroidExtraction):
         return records
 
     def check_indicators(self):
+        if not self.indicators:
+            return
+
         root_packages_path = os.path.join("..", "..", "data", "root_packages.txt")
         root_packages_string = pkg_resources.resource_string(__name__, root_packages_path)
         root_packages = root_packages_string.decode("utf-8").split("\n")
         root_packages = [rp.strip() for rp in root_packages]
 
-
         for result in self.results:
             if result["package_name"] in root_packages:
                 self.log.warning("Found an installed package related to rooting/jailbreaking: \"%s\"",
-                                result["package_name"])
+                                 result["package_name"])
                 self.detected.append(result)
             if result["package_name"] in self.indicators.ioc_app_ids:
                 self.log.warning("Found a malicious package name: \"%s\"",
@@ -92,6 +95,9 @@ class Packages(AndroidExtraction):
         self._adb_connect()
 
         packages = self._adb_command("pm list packages -U -u -i -f")
+        if packages.strip() == "Error: Unknown option: -U":
+            packages = self._adb_command("pm list packages -u -i -f")
+
         for line in packages.split("\n"):
             line = line.strip()
             if not line.startswith("package:"):

mvt/android/modules/adb/processes.py

@@ -9,6 +9,7 @@ from .base import AndroidExtraction
 
 log = logging.getLogger(__name__)
 
+
 class Processes(AndroidExtraction):
     """This module extracts details on running processes."""
 

mvt/android/modules/adb/rootbinaries.py

@@ -12,6 +12,7 @@ from .base import AndroidExtraction
 
 log = logging.getLogger(__name__)
 
+
 class RootBinaries(AndroidExtraction):
     """This module extracts the list of installed packages."""
 

mvt/android/modules/adb/sms.py

@@ -15,12 +15,12 @@ log = logging.getLogger(__name__)
 
 SMS_BUGLE_PATH = "data/data/com.google.android.apps.messaging/databases/bugle_db"
 SMS_BUGLE_QUERY = """
-SELECT 
+SELECT
     ppl.normalized_destination AS number,
     p.timestamp AS timestamp,
-CASE WHEN m.sender_id IN 
+CASE WHEN m.sender_id IN
 (SELECT _id FROM participants WHERE contact_id=-1)
-THEN 2 ELSE 1 END incoming, p.text AS text 
+THEN 2 ELSE 1 END incoming, p.text AS text
 FROM messages m, conversations c, parts p,
         participants ppl, conversation_participants cp
 WHERE (m.conversation_id = c._id)
@@ -31,14 +31,15 @@ WHERE (m.conversation_id = c._id)
 
 SMS_MMSSMS_PATH = "data/data/com.android.providers.telephony/databases/mmssms.db"
 SMS_MMSMS_QUERY = """
-SELECT 
+SELECT
     address AS number,
     date_sent AS timestamp,
     type as incoming,
-    body AS text 
+    body AS text
 FROM sms;
 """
 
+
 class SMS(AndroidExtraction):
     """This module extracts all SMS messages containing links."""
 
@@ -62,7 +63,7 @@ class SMS(AndroidExtraction):
             return
 
         for message in self.results:
-            if not "text" in message:
+            if "text" not in message:
                 continue
 
             message_links = check_for_links(message["text"])
@@ -77,7 +78,7 @@ class SMS(AndroidExtraction):
         """
         conn = sqlite3.connect(db_path)
         cur = conn.cursor()
-        
+
         if (self.SMS_DB_TYPE == 1):
             cur.execute(SMS_BUGLE_QUERY)
         elif (self.SMS_DB_TYPE == 2):

mvt/android/modules/adb/whatsapp.py

@@ -16,6 +16,7 @@ log = logging.getLogger(__name__)
 
 WHATSAPP_PATH = "data/data/com.whatsapp/databases/msgstore.db"
 
+
 class Whatsapp(AndroidExtraction):
     """This module extracts all WhatsApp messages containing links."""
 
@@ -39,7 +40,7 @@ class Whatsapp(AndroidExtraction):
             return
 
         for message in self.results:
-            if not "data" in message:
+            if "data" not in message:
                 continue
 
             message_links = check_for_links(message["data"])

mvt/android/modules/backup/__init__.py

@@ -5,4 +5,4 @@
 
 from .sms import SMS
 
-BACKUP_MODULES = [SMS,]
+BACKUP_MODULES = [SMS]

mvt/android/modules/backup/sms.py

@@ -24,7 +24,7 @@ class SMS(MVTModule):
             return
 
         for message in self.results:
-            if not "body" in message:
+            if "body" not in message:
                 continue
 
             message_links = check_for_links(message["body"])

mvt/common/logo.py

@@ -16,7 +16,7 @@ def logo():
 
     try:
         latest_version = check_for_updates()
-    except:
+    except Exception:
         pass
     else:
         if latest_version:

mvt/common/module.py

@@ -10,27 +10,27 @@ import re
 
 import simplejson as json
 
-from .indicators import Indicators
-
 
 class DatabaseNotFoundError(Exception):
     pass
 
+
 class DatabaseCorruptedError(Exception):
     pass
 
+
 class InsufficientPrivileges(Exception):
     pass
 
+
 class MVTModule(object):
-    """This class provides a base for all extraction modules.
-    """
+    """This class provides a base for all extraction modules."""
 
     enabled = True
     slug = None
 
     def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 fast_mode=False, log=None, results=[]):
+                 fast_mode=False, log=None, results=None):
         """Initialize module.
 
         :param file_path: Path to the module's database file, if there is any
@@ -51,7 +51,7 @@ class MVTModule(object):
         self.fast_mode = fast_mode
         self.log = log
         self.indicators = None
-        self.results = results
+        self.results = results if results else []
         self.detected = []
         self.timeline = []
         self.timeline_detected = []
@@ -66,8 +66,7 @@ class MVTModule(object):
             return cls(results=results, log=log)
 
     def get_slug(self):
-        """Use the module's class name to retrieve a slug
-        """
+        """Use the module's class name to retrieve a slug"""
         if self.slug:
             return self.slug
 
@@ -77,12 +76,13 @@ class MVTModule(object):
     def check_indicators(self):
         """Check the results of this module against a provided list of
         indicators.
+
+
         """
         raise NotImplementedError
 
     def save_to_json(self):
-        """Save the collected results to a json file.
-        """
+        """Save the collected results to a json file."""
         if not self.output_folder:
             return
 
@@ -112,6 +112,7 @@ class MVTModule(object):
         """Serialize entry as JSON to deduplicate repeated entries
 
         :param timeline: List of entries from timeline to deduplicate
+
         """
         timeline_set = set()
         for record in timeline:
@@ -141,8 +142,7 @@ class MVTModule(object):
         self.timeline_detected = self._deduplicate_timeline(self.timeline_detected)
 
     def run(self):
-        """Run the main module procedure.
-        """
+        """Run the main module procedure."""
         raise NotImplementedError
 
 
@@ -190,6 +190,7 @@ def save_timeline(timeline, timeline_path):
 
     :param timeline: List of records to order and store
     :param timeline_path: Path to the csv file to store the timeline to
+
     """
     with io.open(timeline_path, "a+", encoding="utf-8") as handle:
         csvoutput = csv.writer(handle, delimiter=",", quotechar="\"")

mvt/common/url.py

@@ -250,6 +250,7 @@ SHORTENER_DOMAINS = [
     "zz.gd",
 ]
 
+
 class URL:
 
     def __init__(self, url):
@@ -268,11 +269,12 @@ class URL:
         :type url: str
         :returns: Domain name extracted from URL
         :rtype: str
+
         """
         # TODO: Properly handle exception.
         try:
             return get_tld(self.url, as_object=True, fix_protocol=True).parsed_url.netloc.lower().lstrip("www.")
-        except:
+        except Exception:
             return None
 
     def get_top_level(self):
@@ -282,18 +284,22 @@ class URL:
         :type url: str
         :returns: Top-level domain name extracted from URL
         :rtype: str
+
         """
         # TODO: Properly handle exception.
         try:
             return get_tld(self.url, as_object=True, fix_protocol=True).fld.lower()
-        except:
+        except Exception:
             return None
 
     def check_if_shortened(self) -> bool:
         """Check if the URL is among list of shortener services.
 
+
         :returns: True if the URL is shortened, otherwise False
+
         :rtype: bool
+
         """
         if self.domain.lower() in SHORTENER_DOMAINS:
             self.is_shortened = True
@@ -301,8 +307,7 @@ class URL:
         return self.is_shortened
 
     def unshorten(self):
-        """Unshorten the URL by requesting an HTTP HEAD response.
-        """
+        """Unshorten the URL by requesting an HTTP HEAD response."""
         res = requests.head(self.url)
         if str(res.status_code).startswith("30"):
             return res.headers["Location"]

mvt/common/utils.py

@@ -16,6 +16,7 @@ def convert_mactime_to_unix(timestamp, from_2001=True):
     :param from_2001: bool: Whether to (Default value = True)
     :param from_2001: Default value = True)
     :returns: Unix epoch timestamp.
+
     """
     if not timestamp:
         return None
@@ -42,8 +43,9 @@ def convert_chrometime_to_unix(timestamp):
     :param timestamp: Chrome timestamp as int.
     :type timestamp: int
     :returns: Unix epoch timestamp.
+
     """
-    epoch_start = datetime.datetime(1601, 1 , 1)
+    epoch_start = datetime.datetime(1601, 1, 1)
     delta = datetime.timedelta(microseconds=timestamp)
     return epoch_start + delta
 
@@ -55,21 +57,25 @@ def convert_timestamp_to_iso(timestamp):
     :type timestamp: int
     :returns: ISO timestamp string in YYYY-mm-dd HH:MM:SS.ms format.
     :rtype: str
+
     """
     try:
         return timestamp.strftime("%Y-%m-%d %H:%M:%S.%f")
     except Exception:
         return None
 
+
 def check_for_links(text):
     """Checks if a given text contains HTTP links.
 
     :param text: Any provided text.
     :type text: str
     :returns: Search results.
+
     """
     return re.findall("(?P<url>https?://[^\s]+)", text, re.IGNORECASE)
 
+
 def get_sha256_from_file_path(file_path):
     """Calculate the SHA256 hash of a file from a file path.
 
@@ -84,6 +90,7 @@ def get_sha256_from_file_path(file_path):
 
     return sha256_hash.hexdigest()
 
+
 # Note: taken from here:
 # https://stackoverflow.com/questions/57014259/json-dumps-on-dictionary-with-bytes-for-keys
 def keys_bytes_to_string(obj):
@@ -92,6 +99,7 @@ def keys_bytes_to_string(obj):
     :param obj: Object to convert from bytes to string.
     :returns: Object converted to string.
     :rtype: str
+
     """
     new_obj = {}
     if not isinstance(obj, dict):