Bumped version

This way when a detection is logged, the user can know which STIX2
file was matched by the module

.github/workflows/python-package.yml

@@ -16,7 +16,8 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: [3.7, 3.8, 3.9]
+        # python-version: [3.7, 3.8, 3.9]
+        python-version: [3.8, 3.9]
 
     steps:
     - uses: actions/checkout@v2
@@ -27,8 +28,9 @@ jobs:
     - name: Install dependencies
       run: |
         python -m pip install --upgrade pip
-        python -m pip install flake8 pytest safety
+        python -m pip install flake8 pytest safety stix2
         if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
+        python -m pip install .
     - name: Lint with flake8
       run: |
         # stop the build if there are Python syntax errors or undefined names
@@ -37,7 +39,5 @@ jobs:
         flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
     - name: Safety checks
       run: safety check
-
+    - name: Test with pytest
-    # - name: Test with pytest
+      run: pytest
-    #   run: |
-    #     pytest

README.md

@@ -15,15 +15,15 @@ It has been developed and released by the [Amnesty International Security Lab](h
 
 ## Installation
 
-MVT can be installed from sources or from [PyPi](https://pypi.org/project/mvt/) (you will need some dependencies, check the [documentation](https://docs.mvt.re/en/latest/install.html)):
+MVT can be installed from sources or from [PyPi](https://pypi.org/project/mvt/) (you will need some dependencies, check the [documentation](https://docs.mvt.re/en/latest/install/)):
 
 ```
 pip3 install mvt
 ```
 
-Alternatively, you can decide to run MVT and all relevant tools through a [Docker container](https://docs.mvt.re/en/latest/docker.html).
+Alternatively, you can decide to run MVT and all relevant tools through a [Docker container](https://docs.mvt.re/en/latest/docker/).
 
-**Please note:** MVT is best run on Linux or Mac systems. [It does not currently support running natively on Windows.](https://docs.mvt.re/en/latest/install.html#mvt-on-windows)
+**Please note:** MVT is best run on Linux or Mac systems. [It does not currently support running natively on Windows.](https://docs.mvt.re/en/latest/install/#mvt-on-windows)
 
 ## Usage
 
@@ -31,4 +31,4 @@ MVT provides two commands `mvt-ios` and `mvt-android`. [Check out the documentat
 
 ## License
 
-The purpose of MVT is to facilitate the ***consensual forensic analysis*** of devices of those who might be targets of sophisticated mobile spyware attacks, especially members of civil society and marginalized communities. We do not want MVT to enable privacy violations of non-consenting individuals.  In order to achieve this, MVT is released under its own license. [Read more here.](https://docs.mvt.re/en/latest/license.html)
+The purpose of MVT is to facilitate the ***consensual forensic analysis*** of devices of those who might be targets of sophisticated mobile spyware attacks, especially members of civil society and marginalized communities. We do not want MVT to enable privacy violations of non-consenting individuals.  In order to achieve this, MVT is released under its own license. [Read more here.](https://docs.mvt.re/en/latest/license/)

docs/iocs.md

@@ -28,10 +28,19 @@ The `--iocs` option can be invoked multiple times to let MVT import multiple STI
 mvt-ios check-backup --iocs ~/iocs/malware1.stix --iocs ~/iocs/malware2.stix2 /path/to/backup
 ```
 
+It is also possible to load STIX2 files automatically from the environment variable `MVT_STIX2`:
+
+```bash
+export MVT_STIX2="/home/user/IOC1.stix2:/home/user/IOC2.stix2"
+```
+
 ## Known repositories of STIX2 IOCs
 
 - The [Amnesty International investigations repository](https://github.com/AmnestyTech/investigations) contains STIX-formatted IOCs for:
     - [Pegasus](https://en.wikipedia.org/wiki/Pegasus_(spyware)) ([STIX2](https://raw.githubusercontent.com/AmnestyTech/investigations/master/2021-07-18_nso/pegasus.stix2))
-- [This repository](https://github.com/Te-k/stalkerware-indicators) contains IOCs for Android stalkerware including [a STIX MVT-compatible file](https://github.com/Te-k/stalkerware-indicators/blob/master/stalkerware.stix2).
+    - [Predator from Cytrox](https://citizenlab.ca/2021/12/pegasus-vs-predator-dissidents-doubly-infected-iphone-reveals-cytrox-mercenary-spyware/) ([STIX2](https://raw.githubusercontent.com/AmnestyTech/investigations/master/2021-12-16_cytrox/cytrox.stix2))
+- [This repository](https://github.com/Te-k/stalkerware-indicators) contains IOCs for Android stalkerware including [a STIX MVT-compatible file](https://raw.githubusercontent.com/Te-k/stalkerware-indicators/master/stalkerware.stix2).
+
+You can automaticallly download the latest public indicator files with the command `mvt-ios download-iocs` or `mvt-android download-iocs`. These commands download the list of indicators listed [here](https://github.com/mvt-project/mvt/blob/main/public_indicators.json) and store them in the [appdir](https://pypi.org/project/appdirs/) folder. They are then loaded automatically by mvt.
 
 Please [open an issue](https://github.com/mvt-project/mvt/issues/) to suggest new sources of STIX-formatted IOCs.

docs/requirements.txt

@@ -1,4 +1,4 @@
-mkdocs==1.2.1
+mkdocs==1.2.3
 mkdocs-autorefs
 mkdocs-material
 mkdocs-material-extensions

mvt/android/cli.py

@@ -9,8 +9,10 @@ import os
 import click
 from rich.logging import RichHandler
 
-from mvt.common.help import *
+from mvt.common.help import (HELP_MSG_FAST, HELP_MSG_IOC,
-from mvt.common.indicators import Indicators, IndicatorsFileBadFormat
+                             HELP_MSG_LIST_MODULES, HELP_MSG_MODULE,
+                             HELP_MSG_OUTPUT, HELP_MSG_SERIAL)
+from mvt.common.indicators import Indicators, download_indicators_files
 from mvt.common.logo import logo
 from mvt.common.module import run_module, save_timeline
 
@@ -26,6 +28,7 @@ logging.basicConfig(level="INFO", format=LOG_FORMAT, handlers=[
     RichHandler(show_path=False, log_time_format="%X")])
 log = logging.getLogger(__name__)
 
+
 #==============================================================================
 # Main
 #==============================================================================
@@ -104,10 +107,11 @@ def download_apks(ctx, all_apks, virustotal, koodous, all_checks, output, from_f
               default=[], help=HELP_MSG_IOC)
 @click.option("--output", "-o", type=click.Path(exists=False),
               help=HELP_MSG_OUTPUT)
+@click.option("--fast", "-f", is_flag=True, help=HELP_MSG_FAST)
 @click.option("--list-modules", "-l", is_flag=True, help=HELP_MSG_LIST_MODULES)
 @click.option("--module", "-m", help=HELP_MSG_MODULE)
 @click.pass_context
-def check_adb(ctx, iocs, output, list_modules, module, serial):
+def check_adb(ctx, iocs, output, fast, list_modules, module, serial):
     if list_modules:
         log.info("Following is the list of available check-adb modules:")
         for adb_module in ADB_MODULES:
@@ -125,13 +129,7 @@ def check_adb(ctx, iocs, output, list_modules, module, serial):
             ctx.exit(1)
 
     indicators = Indicators(log=log)
-    for ioc_path in iocs:
+    indicators.load_indicators_files(iocs)
-        try:
-            indicators.parse_stix2(ioc_path)
-        except IndicatorsFileBadFormat as e:
-            log.critical(e)
-            ctx.exit(1)
-    log.info("Loaded a total of %d indicators", indicators.ioc_count)
 
     timeline = []
     timeline_detected = []
@@ -139,14 +137,14 @@ def check_adb(ctx, iocs, output, list_modules, module, serial):
         if module and adb_module.__name__ != module:
             continue
 
-        m = adb_module(output_folder=output, log=logging.getLogger(adb_module.__module__))
+        m = adb_module(output_folder=output, fast_mode=fast,
+                       log=logging.getLogger(adb_module.__module__))
+        if indicators.total_ioc_count:
+            m.indicators = indicators
+            m.indicators.log = m.log
         if serial:
             m.serial = serial
 
-        if iocs:
-            indicators.log = m.log
-            m.indicators = indicators
-
         run_module(m)
         timeline.extend(m.timeline)
         timeline_detected.extend(m.timeline_detected)
@@ -179,31 +177,97 @@ def check_backup(ctx, iocs, output, backup_path, serial):
             ctx.exit(1)
 
     indicators = Indicators(log=log)
-    for ioc_path in iocs:
+    indicators.load_indicators_files(iocs)
-        try:
-            indicators.parse_stix2(ioc_path)
-        except IndicatorsFileBadFormat as e:
-            log.critical(e)
-            ctx.exit(1)
-    log.info("Loaded a total of %d indicators", indicators.ioc_count)
 
     if os.path.isfile(backup_path):
         log.critical("The path you specified is a not a folder!")
 
         if os.path.basename(backup_path) == "backup.ab":
-            log.info("You can use ABE (https://github.com/nelenkov/android-backup-extractor) " \
+            log.info("You can use ABE (https://github.com/nelenkov/android-backup-extractor) "
                      "to extract 'backup.ab' files!")
         ctx.exit(1)
 
     for module in BACKUP_MODULES:
         m = module(base_folder=backup_path, output_folder=output,
                    log=logging.getLogger(module.__module__))
-
+        if indicators.total_ioc_count:
+            m.indicators = indicators
+            m.indicators.log = m.log
         if serial:
             m.serial = serial
 
-        if iocs:
-            indicators.log = m.log
-            m.indicators = indicators
-
         run_module(m)
+
+
+#==============================================================================
+# Command: check-iocs
+#==============================================================================
+@cli.command("check-iocs", help="Compare stored JSON results to provided indicators")
+@click.option("--iocs", "-i", type=click.Path(exists=True), multiple=True,
+              default=[], help=HELP_MSG_IOC)
+@click.option("--list-modules", "-l", is_flag=True, help=HELP_MSG_LIST_MODULES)
+@click.option("--module", "-m", help=HELP_MSG_MODULE)
+@click.argument("FOLDER", type=click.Path(exists=True))
+@click.pass_context
+def check_iocs(ctx, iocs, list_modules, module, folder):
+    all_modules = []
+    for entry in BACKUP_MODULES + ADB_MODULES:
+        if entry not in all_modules:
+            all_modules.append(entry)
+
+    if list_modules:
+        log.info("Following is the list of available check-iocs modules:")
+        for iocs_module in all_modules:
+            log.info(" - %s", iocs_module.__name__)
+
+        return
+
+    log.info("Checking stored results against provided indicators...")
+
+    indicators = Indicators(log=log)
+    indicators.load_indicators_files(iocs)
+
+    total_detections = 0
+    for file_name in os.listdir(folder):
+        name_only, ext = os.path.splitext(file_name)
+        file_path = os.path.join(folder, file_name)
+
+        # TODO: Skipping processing of result files that are not json.
+        #       We might want to revisit this eventually.
+        if ext != ".json":
+            continue
+
+        for iocs_module in all_modules:
+            if module and iocs_module.__name__ != module:
+                continue
+
+            if iocs_module().get_slug() != name_only:
+                continue
+
+            log.info("Loading results from \"%s\" with module %s", file_name,
+                     iocs_module.__name__)
+
+            m = iocs_module.from_json(file_path,
+                                      log=logging.getLogger(iocs_module.__module__))
+            if indicators.total_ioc_count > 0:
+                m.indicators = indicators
+                m.indicators.log = m.log
+
+            try:
+                m.check_indicators()
+            except NotImplementedError:
+                continue
+            else:
+                total_detections += len(m.detected)
+
+    if total_detections > 0:
+        log.warning("The check of the results produced %d detections!",
+                    total_detections)
+
+
+#==============================================================================
+# Command: download-iocs
+#==============================================================================
+@cli.command("download-iocs", help="Download public STIX2 indicators")
+def download_indicators():
+    download_indicators_files(log)

mvt/android/download_apks.py

@@ -7,7 +7,6 @@ import json
 import logging
 import os
 
-import pkg_resources
 from tqdm import tqdm
 
 from mvt.common.module import InsufficientPrivileges
@@ -17,6 +16,7 @@ from .modules.adb.packages import Packages
 
 log = logging.getLogger(__name__)
 
+
 # TODO: Would be better to replace tqdm with rich.progress to reduce
 #       the number of dependencies. Need to investigate whether
 #       it's possible to have a similar callback system.
@@ -138,7 +138,7 @@ class DownloadAPKs(AndroidExtraction):
                     packages_selection.append(package)
 
             log.info("Selected only %d packages which are not marked as system",
-                len(packages_selection))
+                     len(packages_selection))
 
         if len(packages_selection) == 0:
             log.info("No packages were selected for download")

mvt/android/lookups/koodous.py

@@ -13,6 +13,7 @@ from rich.text import Text
 
 log = logging.getLogger(__name__)
 
+
 def koodous_lookup(packages):
     log.info("Looking up all extracted files on Koodous (www.koodous.com)")
     log.info("This might take a while...")

mvt/android/lookups/virustotal.py

@@ -13,6 +13,7 @@ from rich.text import Text
 
 log = logging.getLogger(__name__)
 
+
 def get_virustotal_report(hashes):
     apikey = "233f22e200ca5822bd91103043ccac138b910db79f29af5616a9afe8b6f215ad"
     url = f"https://www.virustotal.com/partners/sysinternals/file-reports?apikey={apikey}"
@@ -36,7 +37,12 @@ def get_virustotal_report(hashes):
         log.error("Unexpected response from VirusTotal: %s", res.status_code)
         return None
 
+
 def virustotal_lookup(packages):
+    # NOTE: This is temporary, until we resolved the issue.
+    log.error("Unfortunately VirusTotal lookup is disabled until further notice, due to unresolved issues with the API service.")
+    return
+
     log.info("Looking up all extracted files on VirusTotal (www.virustotal.com)")
 
     unique_hashes = []
@@ -48,6 +54,7 @@ def virustotal_lookup(packages):
     total_unique_hashes = len(unique_hashes)
 
     detections = {}
+
     def virustotal_query(batch):
         report = get_virustotal_report(batch)
         if not report:

mvt/android/modules/adb/__init__.py

@@ -4,20 +4,23 @@
 #   https://license.mvt.re/1.1/
 
 from .chrome_history import ChromeHistory
+from .dumpsys_accessibility import DumpsysAccessibility
 from .dumpsys_batterystats import DumpsysBatterystats
 from .dumpsys_full import DumpsysFull
 from .dumpsys_packages import DumpsysPackages
 from .dumpsys_procstats import DumpsysProcstats
 from .dumpsys_receivers import DumpsysReceivers
 from .files import Files
+from .getprop import Getprop
 from .logcat import Logcat
 from .packages import Packages
 from .processes import Processes
-from .rootbinaries import RootBinaries
+from .root_binaries import RootBinaries
+from .settings import Settings
 from .sms import SMS
 from .whatsapp import Whatsapp
 
-ADB_MODULES = [ChromeHistory, SMS, Whatsapp, Processes,
+ADB_MODULES = [ChromeHistory, SMS, Whatsapp, Processes, Getprop, Settings,
-               DumpsysBatterystats, DumpsysProcstats,
+               DumpsysAccessibility, DumpsysBatterystats, DumpsysProcstats,
                DumpsysPackages, DumpsysReceivers, DumpsysFull,
                Packages, RootBinaries, Logcat, Files]

mvt/android/modules/adb/base.py

@@ -15,7 +15,7 @@ from adb_shell.adb_device import AdbDeviceTcp, AdbDeviceUsb
 from adb_shell.auth.keygen import keygen, write_public_keyfile
 from adb_shell.auth.sign_pythonrsa import PythonRSASigner
 from adb_shell.exceptions import (AdbCommandFailureException, DeviceAuthError,
-                                  UsbReadFailedError)
+                                  UsbDeviceNotFoundError, UsbReadFailedError)
 from usb1 import USBErrorAccess, USBErrorBusy
 
 from mvt.common.module import InsufficientPrivileges, MVTModule
@@ -25,6 +25,7 @@ log = logging.getLogger(__name__)
 ADB_KEY_PATH = os.path.expanduser("~/.android/adbkey")
 ADB_PUB_KEY_PATH = os.path.expanduser("~/.android/adbkey.pub")
 
+
 class AndroidExtraction(MVTModule):
     """This class provides a base for all Android extraction modules."""
 
@@ -64,7 +65,11 @@ class AndroidExtraction(MVTModule):
         # If no serial was specified or if the serial does not seem to be
         # a HOST:PORT definition, we use the USB transport.
         if not self.serial or ":" not in self.serial:
-            self.device = AdbDeviceUsb(serial=self.serial)
+            try:
+                self.device = AdbDeviceUsb(serial=self.serial)
+            except UsbDeviceNotFoundError:
+                log.critical("No device found. Make sure it is connected and unlocked.")
+                sys.exit(-1)
         # Otherwise we try to use the TCP transport.
         else:
             addr = self.serial.split(":")
@@ -89,7 +94,7 @@ class AndroidExtraction(MVTModule):
             except OSError as e:
                 if e.errno == 113 and self.serial:
                     log.critical("Unable to connect to the device %s: did you specify the correct IP addres?",
-                              self.serial)
+                                 self.serial)
                     sys.exit(-1)
             else:
                 break
@@ -111,7 +116,7 @@ class AndroidExtraction(MVTModule):
         :returns: Output of command
 
         """
-        return self.device.shell(command)
+        return self.device.shell(command, read_timeout_s=200.0)
 
     def _adb_check_if_root(self):
         """Check if we have a `su` binary on the Android device.

mvt/android/modules/adb/chrome_history.py

@@ -16,6 +16,7 @@ log = logging.getLogger(__name__)
 
 CHROME_HISTORY_PATH = "data/data/com.android.chrome/app_chrome/Default/History"
 
+
 class ChromeHistory(AndroidExtraction):
     """This module extracts records from Android's Chrome browsing history."""
 

mvt/android/modules/adb/dumpsys_accessibility.py

@@ -0,0 +1,53 @@
+# Mobile Verification Toolkit (MVT)
+# Copyright (c) 2021 The MVT Project Authors.
+# Use of this software is governed by the MVT License 1.1 that can be found at
+#   https://license.mvt.re/1.1/
+
+import io
+import logging
+import os
+
+from .base import AndroidExtraction
+
+log = logging.getLogger(__name__)
+
+
+class DumpsysAccessibility(AndroidExtraction):
+    """This module extracts stats on accessibility."""
+
+    def __init__(self, file_path=None, base_folder=None, output_folder=None,
+                 serial=None, fast_mode=False, log=None, results=[]):
+        super().__init__(file_path=file_path, base_folder=base_folder,
+                         output_folder=output_folder, fast_mode=fast_mode,
+                         log=log, results=results)
+
+    def run(self):
+        self._adb_connect()
+
+        stats = self._adb_command("dumpsys accessibility")
+
+        in_services = False
+        for line in stats.split("\n"):
+            if line.strip().startswith("installed services:"):
+                in_services = True
+                continue
+
+            if not in_services:
+                continue
+
+            if line.strip() == "}":
+                break
+
+            service = line.split(":")[1].strip()
+            log.info("Found installed accessibility service \"%s\"", service)
+
+        if self.output_folder:
+            acc_path = os.path.join(self.output_folder,
+                                      "dumpsys_accessibility.txt")
+            with io.open(acc_path, "w", encoding="utf-8") as handle:
+                handle.write(stats)
+
+            log.info("Records from dumpsys accessibility stored at %s",
+                     acc_path)
+
+        self._adb_disconnect()

mvt/android/modules/adb/dumpsys_batterystats.py

@@ -10,6 +10,7 @@ from .base import AndroidExtraction
 
 log = logging.getLogger(__name__)
 
+
 class DumpsysBatterystats(AndroidExtraction):
     """This module extracts stats on battery consumption by processes."""
 
@@ -30,7 +31,7 @@ class DumpsysBatterystats(AndroidExtraction):
                 handle.write(stats)
 
             log.info("Records from dumpsys batterystats stored at %s",
-                      stats_path)
+                     stats_path)
 
         history = self._adb_command("dumpsys batterystats --history")
         if self.output_folder:

mvt/android/modules/adb/dumpsys_full.py

@@ -10,6 +10,7 @@ from .base import AndroidExtraction
 
 log = logging.getLogger(__name__)
 
+
 class DumpsysFull(AndroidExtraction):
     """This module extracts stats on battery consumption by processes."""
 
@@ -30,6 +31,6 @@ class DumpsysFull(AndroidExtraction):
                 handle.write(stats)
 
             log.info("Full dumpsys output stored at %s",
-                      stats_path)
+                     stats_path)
 
         self._adb_disconnect()

mvt/android/modules/adb/dumpsys_procstats.py

@@ -10,6 +10,7 @@ from .base import AndroidExtraction
 
 log = logging.getLogger(__name__)
 
+
 class DumpsysProcstats(AndroidExtraction):
     """This module extracts stats on memory consumption by processes."""
 

mvt/android/modules/adb/dumpsys_receivers.py

@@ -4,16 +4,17 @@
 #   https://license.mvt.re/1.1/
 
 import logging
-import os
 
 from .base import AndroidExtraction
 
 log = logging.getLogger(__name__)
 
-ACTION_NEW_OUTGOING_SMS = "android.provider.Telephony.NEW_OUTGOING_SMS"
+INTENT_NEW_OUTGOING_SMS = "android.provider.Telephony.NEW_OUTGOING_SMS"
-ACTION_SMS_RECEIVED = "android.provider.Telephony.SMS_RECEIVED"
+INTENT_SMS_RECEIVED = "android.provider.Telephony.SMS_RECEIVED"
-ACTION_DATA_SMS_RECEIVED = "android.intent.action.DATA_SMS_RECEIVED"
+INTENT_DATA_SMS_RECEIVED = "android.intent.action.DATA_SMS_RECEIVED"
-ACTION_PHONE_STATE = "android.intent.action.PHONE_STATE"
+INTENT_PHONE_STATE = "android.intent.action.PHONE_STATE"
+INTENT_NEW_OUTGOING_CALL = "android.intent.action.NEW_OUTGOING_CALL"
+
 
 class DumpsysReceivers(AndroidExtraction):
     """This module extracts details on receivers for risky activities."""
@@ -24,6 +25,24 @@ class DumpsysReceivers(AndroidExtraction):
                          output_folder=output_folder, fast_mode=fast_mode,
                          log=log, results=results)
 
+    def check_indicators(self):
+        for result in self.results:
+            if result["activity"] == INTENT_NEW_OUTGOING_SMS:
+                self.log.info("Found a receiver to intercept outgoing SMS messages: \"%s\"",
+                              result["receiver"])
+            elif result["activity"] == INTENT_SMS_RECEIVED:
+                self.log.info("Found a receiver to intercept incoming SMS messages: \"%s\"",
+                              result["receiver"])
+            elif result["activity"] == INTENT_DATA_SMS_RECEIVED:
+                self.log.info("Found a receiver to intercept incoming data SMS message: \"%s\"",
+                              result["receiver"])
+            elif result["activity"] == INTENT_PHONE_STATE:
+                self.log.info("Found a receiver monitoring telephony state/incoming calls: \"%s\"",
+                              result["receiver"])
+            elif result["activity"] == INTENT_NEW_OUTGOING_CALL:
+                self.log.info("Found a receiver monitoring outgoing calls: \"%s\"",
+                              result["receiver"])
+
     def run(self):
         self._adb_connect()
 
@@ -34,17 +53,20 @@ class DumpsysReceivers(AndroidExtraction):
         activity = None
         for line in output.split("\n"):
             # Find activity block markers.
-            if line.strip().startswith(ACTION_NEW_OUTGOING_SMS):
+            if line.strip().startswith(INTENT_NEW_OUTGOING_SMS):
-                activity = ACTION_NEW_OUTGOING_SMS
+                activity = INTENT_NEW_OUTGOING_SMS
                 continue
-            elif line.strip().startswith(ACTION_SMS_RECEIVED):
+            elif line.strip().startswith(INTENT_SMS_RECEIVED):
-                activity = ACTION_SMS_RECEIVED
+                activity = INTENT_SMS_RECEIVED
                 continue
-            elif line.strip().startswith(ACTION_PHONE_STATE):
+            elif line.strip().startswith(INTENT_PHONE_STATE):
-                activity = ACTION_PHONE_STATE
+                activity = INTENT_PHONE_STATE
                 continue
-            elif line.strip().startswith(ACTION_DATA_SMS_RECEIVED):
+            elif line.strip().startswith(INTENT_DATA_SMS_RECEIVED):
-                activity = ACTION_DATA_SMS_RECEIVED
+                activity = INTENT_DATA_SMS_RECEIVED
+                continue
+            elif line.strip().startswith(INTENT_NEW_OUTGOING_CALL):
+                activity = INTENT_NEW_OUTGOING_CALL
                 continue
 
             # If we are not in an activity block yet, skip.
@@ -65,19 +87,6 @@ class DumpsysReceivers(AndroidExtraction):
             if package_name == "com.google.android.gms":
                 continue
 
-            if activity == ACTION_NEW_OUTGOING_SMS:
-                self.log.info("Found a receiver to intercept outgoing SMS messages: \"%s\"",
-                    receiver)
-            elif activity == ACTION_SMS_RECEIVED:
-                self.log.info("Found a receiver to intercept incoming SMS messages: \"%s\"",
-                    receiver)
-            elif activity == ACTION_DATA_SMS_RECEIVED:
-                self.log.info("Found a receiver to intercept incoming data SMS message: \"%s\"",
-                    receiver)
-            elif activity == ACTION_PHONE_STATE:
-                self.log.info("Found a receiver monitoring telephony state: \"%s\"",
-                    receiver)
-
             self.results.append({
                 "activity": activity,
                 "package_name": package_name,

mvt/android/modules/adb/files.py

@@ -3,31 +3,95 @@
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 
+import datetime
 import logging
-import os
+import stat
+
+from mvt.common.utils import convert_timestamp_to_iso
 
 from .base import AndroidExtraction
 
 log = logging.getLogger(__name__)
 
+
 class Files(AndroidExtraction):
-    """This module extracts the list of installed packages."""
+    """This module extracts the list of files on the device."""
 
     def __init__(self, file_path=None, base_folder=None, output_folder=None,
                  serial=None, fast_mode=False, log=None, results=[]):
         super().__init__(file_path=file_path, base_folder=base_folder,
                          output_folder=output_folder, fast_mode=fast_mode,
                          log=log, results=results)
+        self.full_find = False
+
+    def find_files(self, file_path):
+        if self.full_find:
+            output = self._adb_command(f"find '{file_path}' -printf '%T@ %m %s %u %g %p\n' 2> /dev/null")
+
+            for file_line in output.splitlines():
+                [unix_timestamp, mode, size, owner, group, full_path] = file_line.rstrip().split(" ", 5)
+                mod_time = convert_timestamp_to_iso(datetime.datetime.utcfromtimestamp(int(float(unix_timestamp))))
+                self.results.append({
+                    "path": full_path,
+                    "modified_time": mod_time,
+                    "mode": mode,
+                    "is_suid": (int(mode, 8) & stat.S_ISUID) == 2048,
+                    "is_sgid": (int(mode, 8) & stat.S_ISGID) == 1024,
+                    "size": size,
+                    "owner": owner,
+                    "group": group,
+                })
+        else:
+            output = self._adb_command(f"find '{file_path}' 2> /dev/null")
+            for file_line in output.splitlines():
+                self.results.append({"path": file_line.rstrip()})
+
+    def serialize(self, record):
+        if "modified_time" in record:
+            return {
+                "timestamp": record["modified_time"],
+                "module": self.__class__.__name__,
+                "event": "file_modified",
+                "data": record["path"],
+            }
+
+    def check_suspicious(self):
+        """Check for files with suspicious permissions"""
+        for result in sorted(self.results, key=lambda item: item["path"]):
+            if result.get("is_suid"):
+                self.log.warning("Found an SUID file in a non-standard directory \"%s\".",
+                                 result["path"])
+                self.detected.append(result)
+
+    def check_indicators(self):
+        """Check file list for known suspicious files or suspicious properties"""
+        self.check_suspicious()
+
+        if not self.indicators:
+            return
+
+        for result in self.results:
+            if self.indicators.check_file_path(result["path"]):
+                self.log.warning("Found a known suspicous file at path: \"%s\"", result["path"])
+                self.detected.append(result)
 
     def run(self):
         self._adb_connect()
 
-        output = self._adb_command("find / -type f 2> /dev/null")
+        output = self._adb_command("find '/' -maxdepth 1 -printf '%T@ %m %s %u %g %p\n' 2> /dev/null")
-        if output and self.output_folder:
+        if output or output.strip().splitlines():
-            files_txt_path = os.path.join(self.output_folder, "files.txt")
+            self.full_find = True
-            with open(files_txt_path, "w") as handle:
-                handle.write(output)
 
-            log.info("List of visible files stored at %s", files_txt_path)
+        for data_path in ["/data/local/tmp/", "/sdcard/", "/tmp/"]:
+            self.find_files(data_path)
+
+        self.log.info("Found %s files in primary Android data directories", len(self.results))
+
+        if self.fast_mode:
+            self.log.info("Flag --fast was enabled: skipping full file listing")
+        else:
+            self.log.info("Processing full file listing. This may take a while...")
+            self.find_files("/")
+            self.log.info("Found %s total files", len(self.results))
 
         self._adb_disconnect()

mvt/android/modules/adb/getprop.py

@@ -0,0 +1,46 @@
+# Mobile Verification Toolkit (MVT)
+# Copyright (c) 2021 The MVT Project Authors.
+# Use of this software is governed by the MVT License 1.1 that can be found at
+#   https://license.mvt.re/1.1/
+
+import logging
+import os
+import re
+
+from .base import AndroidExtraction
+
+log = logging.getLogger(__name__)
+
+
+class Getprop(AndroidExtraction):
+    """This module extracts device properties from getprop command."""
+
+    def __init__(self, file_path=None, base_folder=None, output_folder=None,
+                 serial=None, fast_mode=False, log=None, results=[]):
+        super().__init__(file_path=file_path, base_folder=base_folder,
+                         output_folder=output_folder, fast_mode=fast_mode,
+                         log=log, results=results)
+
+        self.results = {} if not results else results
+
+    def run(self):
+        self._adb_connect()
+
+        rxp = re.compile("\\[(.+?)\\]: \\[(.+?)\\]")
+        out = self._adb_command("getprop")
+        for line in out.splitlines():
+            line = line.strip()
+            if line == "":
+                continue
+
+            matches = re.findall(rxp, line)
+            if not matches or len(matches[0]) != 2:
+                continue
+
+            key = matches[0][0]
+            value = matches[0][1]
+            self.results[key] = value
+
+        self._adb_disconnect()
+
+        self.log.info("Extracted %d Android system properties", len(self.results))

mvt/android/modules/adb/packages.py

@@ -8,10 +8,14 @@ import os
 
 import pkg_resources
 
+from mvt.android.lookups.koodous import koodous_lookup
+from mvt.android.lookups.virustotal import virustotal_lookup
+
 from .base import AndroidExtraction
 
 log = logging.getLogger(__name__)
 
+
 class Packages(AndroidExtraction):
     """This module extracts the list of installed packages."""
 
@@ -41,30 +45,29 @@ class Packages(AndroidExtraction):
         return records
 
     def check_indicators(self):
-        if not self.indicators:
-            return
-
         root_packages_path = os.path.join("..", "..", "data", "root_packages.txt")
         root_packages_string = pkg_resources.resource_string(__name__, root_packages_path)
         root_packages = root_packages_string.decode("utf-8").split("\n")
         root_packages = [rp.strip() for rp in root_packages]
 
-
         for result in self.results:
             if result["package_name"] in root_packages:
                 self.log.warning("Found an installed package related to rooting/jailbreaking: \"%s\"",
-                                result["package_name"])
-                self.detected.append(result)
-            if result["package_name"] in self.indicators.ioc_app_ids:
-                self.log.warning("Found a malicious package name: \"%s\"",
                                  result["package_name"])
                 self.detected.append(result)
-                for file in result["files"]:
+                continue
-                    if file["sha256"] in self.indicators.ioc_files_sha256:
+
-                        self.log.warning("Found a malicious APK: \"%s\" %s",
+            ioc = self.indicators.check_app_id(result.get("package_name"))
-                                         result["package_name"],
+            if ioc:
-                                         file["sha256"])
+                result["matched_indicators"] = ioc
-                        self.detected.append(result)
+                self.detected.append(result)
+                continue
+
+            for package_file in result["files"]:
+                ioc = self.indicators.check_file_hash(package_file["sha256"])
+                if ioc:
+                    result["matched_indicators"] = ioc
+                    self.detected.append(result)
 
     def _get_files_for_package(self, package_name):
         output = self._adb_command(f"pm path {package_name}")
@@ -95,6 +98,9 @@ class Packages(AndroidExtraction):
         self._adb_connect()
 
         packages = self._adb_command("pm list packages -U -u -i -f")
+        if packages.strip() == "Error: Unknown option: -U":
+            packages = self._adb_command("pm list packages -u -i -f")
+
         for line in packages.split("\n"):
             line = line.strip()
             if not line.startswith("package:"):
@@ -154,13 +160,19 @@ class Packages(AndroidExtraction):
                     if result["package_name"] == package_name:
                         self.results[i][cmd["field"]] = True
 
+        packages_to_lookup = []
         for result in self.results:
             if result["system"]:
                 continue
 
+            packages_to_lookup.append(result)
             self.log.info("Found non-system package with name \"%s\" installed by \"%s\" on %s",
                           result["package_name"], result["installer"], result["timestamp"])
 
+        if not self.fast_mode:
+            virustotal_lookup(packages_to_lookup)
+            koodous_lookup(packages_to_lookup)
+
         self.log.info("Extracted at total of %d installed package names",
                       len(self.results))
 

mvt/android/modules/adb/processes.py

@@ -9,6 +9,7 @@ from .base import AndroidExtraction
 
 log = logging.getLogger(__name__)
 
+
 class Processes(AndroidExtraction):
     """This module extracts details on running processes."""
 

mvt/android/modules/adb/root_binaries.py

@@ -12,6 +12,7 @@ from .base import AndroidExtraction
 
 log = logging.getLogger(__name__)
 
+
 class RootBinaries(AndroidExtraction):
     """This module extracts the list of installed packages."""
 

mvt/android/modules/adb/settings.py

@@ -0,0 +1,106 @@
+# Mobile Verification Toolkit (MVT)
+# Copyright (c) 2021 The MVT Project Authors.
+# Use of this software is governed by the MVT License 1.1 that can be found at
+#   https://license.mvt.re/1.1/
+
+import logging
+import os
+import re
+
+from .base import AndroidExtraction
+
+log = logging.getLogger(__name__)
+
+
+ANDROID_DANGEROUS_SETTINGS = [
+    {
+        "description": "disabled Google Play Services apps verification",
+        "key": "verifier_verify_adb_installs",
+        "safe_value": "1",
+    },
+    {
+        "description": "disabled Google Play Protect",
+        "key": "package_verifier_enable",
+        "safe_value": "1",
+    },
+    {
+        "description": "disabled Google Play Protect",
+        "key": "package_verifier_user_consent",
+        "safe_value": "1",
+    },
+    {
+        "description": "disabled Google Play Protect",
+        "key": "upload_apk_enable",
+        "safe_value": "1",
+    },
+    {
+        "description": "enabled installation of non-market apps",
+        "key": "install_non_market_apps",
+        "safe_value": "0",
+    },
+    {
+        "description": "disabled confirmation of adb apps installation",
+        "key": "adb_install_need_confirm",
+        "safe_value": "1",
+    },
+    {
+        "description": "disabled sharing of security reports",
+        "key": "send_security_reports",
+        "safe_value": "1",
+    },
+    {
+        "description": "disabled sharing of crash logs with manufacturer",
+        "key": "samsung_errorlog_agree",
+        "safe_value": "1",
+    },
+    {
+        "description": "disabled applications errors reports",
+        "key": "send_action_app_error",
+        "safe_value": "1",
+    },
+]
+
+class Settings(AndroidExtraction):
+    """This module extracts Android system settings."""
+
+    def __init__(self, file_path=None, base_folder=None, output_folder=None,
+                 serial=None, fast_mode=False, log=None, results=[]):
+        super().__init__(file_path=file_path, base_folder=base_folder,
+                         output_folder=output_folder, fast_mode=fast_mode,
+                         log=log, results=results)
+
+        self.results = {} if not results else results
+
+    def check_indicators(self):
+        for namespace, settings in self.results.items():
+            for key, value in settings.items():
+                for danger in ANDROID_DANGEROUS_SETTINGS:
+                    # Check if one of the dangerous settings is using an unsafe
+                    # value (different than the one specified).
+                    if danger["key"] == key and danger["safe_value"] != value:
+                        self.log.warning("Found suspicious setting \"%s = %s\" (%s)",
+                                         key, value, danger["description"])
+                        break
+
+    def run(self):
+        self._adb_connect()
+
+        for namespace in ["system", "secure", "global"]:
+            out = self._adb_command(f"cmd settings list {namespace}")
+            if not out:
+                continue
+
+            self.results[namespace] = {}
+
+            for line in out.splitlines():
+                line = line.strip()
+                if line == "":
+                    continue
+
+                fields = line.split("=", 1)
+                try:
+                    self.results[namespace][fields[0]] = fields[1]
+                except IndexError:
+                    continue
+
+        self._adb_disconnect()

mvt/android/modules/adb/sms.py

@@ -15,12 +15,12 @@ log = logging.getLogger(__name__)
 
 SMS_BUGLE_PATH = "data/data/com.google.android.apps.messaging/databases/bugle_db"
 SMS_BUGLE_QUERY = """
-SELECT 
+SELECT
     ppl.normalized_destination AS number,
     p.timestamp AS timestamp,
-CASE WHEN m.sender_id IN 
+CASE WHEN m.sender_id IN
 (SELECT _id FROM participants WHERE contact_id=-1)
-THEN 2 ELSE 1 END incoming, p.text AS text 
+THEN 2 ELSE 1 END incoming, p.text AS text
 FROM messages m, conversations c, parts p,
         participants ppl, conversation_participants cp
 WHERE (m.conversation_id = c._id)
@@ -31,14 +31,15 @@ WHERE (m.conversation_id = c._id)
 
 SMS_MMSSMS_PATH = "data/data/com.android.providers.telephony/databases/mmssms.db"
 SMS_MMSMS_QUERY = """
-SELECT 
+SELECT
     address AS number,
     date_sent AS timestamp,
     type as incoming,
-    body AS text 
+    body AS text
 FROM sms;
 """
 
+
 class SMS(AndroidExtraction):
     """This module extracts all SMS messages containing links."""
 
@@ -62,7 +63,7 @@ class SMS(AndroidExtraction):
             return
 
         for message in self.results:
-            if not "text" in message:
+            if "text" not in message:
                 continue
 
             message_links = check_for_links(message["text"])
@@ -77,7 +78,7 @@ class SMS(AndroidExtraction):
         """
         conn = sqlite3.connect(db_path)
         cur = conn.cursor()
-        
+
         if (self.SMS_DB_TYPE == 1):
             cur.execute(SMS_BUGLE_QUERY)
         elif (self.SMS_DB_TYPE == 2):

mvt/android/modules/adb/whatsapp.py

@@ -16,6 +16,7 @@ log = logging.getLogger(__name__)
 
 WHATSAPP_PATH = "data/data/com.whatsapp/databases/msgstore.db"
 
+
 class Whatsapp(AndroidExtraction):
     """This module extracts all WhatsApp messages containing links."""
 
@@ -39,7 +40,7 @@ class Whatsapp(AndroidExtraction):
             return
 
         for message in self.results:
-            if not "data" in message:
+            if "data" not in message:
                 continue
 
             message_links = check_for_links(message["data"])

mvt/android/modules/backup/__init__.py

@@ -5,4 +5,4 @@
 
 from .sms import SMS
 
-BACKUP_MODULES = [SMS,]
+BACKUP_MODULES = [SMS]

mvt/android/modules/backup/sms.py

@@ -24,7 +24,7 @@ class SMS(MVTModule):
             return
 
         for message in self.results:
-            if not "body" in message:
+            if "body" not in message:
                 continue
 
             message_links = check_for_links(message["body"])

mvt/common/logo.py

@@ -16,7 +16,7 @@ def logo():
 
     try:
         latest_version = check_for_updates()
-    except:
+    except Exception:
         pass
     else:
         if latest_version:

mvt/common/module.py

@@ -10,18 +10,19 @@ import re
 
 import simplejson as json
 
-from .indicators import Indicators
-
 
 class DatabaseNotFoundError(Exception):
     pass
 
+
 class DatabaseCorruptedError(Exception):
     pass
 
+
 class InsufficientPrivileges(Exception):
     pass
 
+
 class MVTModule(object):
     """This class provides a base for all extraction modules."""
 
@@ -29,7 +30,7 @@ class MVTModule(object):
     slug = None
 
     def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 fast_mode=False, log=None, results=[]):
+                 fast_mode=False, log=None, results=None):
         """Initialize module.
 
         :param file_path: Path to the module's database file, if there is any
@@ -50,7 +51,7 @@ class MVTModule(object):
         self.fast_mode = fast_mode
         self.log = log
         self.indicators = None
-        self.results = results
+        self.results = results if results else []
         self.detected = []
         self.timeline = []
         self.timeline_detected = []

mvt/common/url.py

@@ -250,6 +250,7 @@ SHORTENER_DOMAINS = [
     "zz.gd",
 ]
 
+
 class URL:
 
     def __init__(self, url):
@@ -273,7 +274,7 @@ class URL:
         # TODO: Properly handle exception.
         try:
             return get_tld(self.url, as_object=True, fix_protocol=True).parsed_url.netloc.lower().lstrip("www.")
-        except:
+        except Exception:
             return None
 
     def get_top_level(self):
@@ -288,7 +289,7 @@ class URL:
         # TODO: Properly handle exception.
         try:
             return get_tld(self.url, as_object=True, fix_protocol=True).fld.lower()
-        except:
+        except Exception:
             return None
 
     def check_if_shortened(self) -> bool:

mvt/common/utils.py

@@ -45,7 +45,7 @@ def convert_chrometime_to_unix(timestamp):
     :returns: Unix epoch timestamp.
 
     """
-    epoch_start = datetime.datetime(1601, 1 , 1)
+    epoch_start = datetime.datetime(1601, 1, 1)
     delta = datetime.timedelta(microseconds=timestamp)
     return epoch_start + delta
 
@@ -64,6 +64,7 @@ def convert_timestamp_to_iso(timestamp):
     except Exception:
         return None
 
+
 def check_for_links(text):
     """Checks if a given text contains HTTP links.
 
@@ -74,6 +75,7 @@ def check_for_links(text):
     """
     return re.findall("(?P<url>https?://[^\s]+)", text, re.IGNORECASE)
 
+
 def get_sha256_from_file_path(file_path):
     """Calculate the SHA256 hash of a file from a file path.
 
@@ -88,6 +90,7 @@ def get_sha256_from_file_path(file_path):
 
     return sha256_hash.hexdigest()
 
+
 # Note: taken from here:
 # https://stackoverflow.com/questions/57014259/json-dumps-on-dictionary-with-bytes-for-keys
 def keys_bytes_to_string(obj):