Merge branch 'feature/android-sub-module-loading' into local-timeline-fixes

Most other Android timestamps appear to be local time. The
results timeline is more useful if all the timestamps
are consistent. I would prefer to use UTC, but that would
mean converting all the other timestamps to UTC as well. We probably
do not have sufficient information to do that accurately,
especially if the device is moving between timezones..

.github/workflows/update-ios-data.yml

@@ -21,7 +21,6 @@ jobs:
           title: '[auto] Update iOS releases and versions'
           commit-message: Add new iOS versions and build numbers
           branch: auto/add-new-ios-releases
-          draft: true
           body: |
             This is an automated pull request to update the iOS releases and version numbers.
           add-paths: |

CONTRIBUTING.md

@@ -1,65 +1,19 @@
-# Contributing to Mobile Verification Toolkit (MVT)
+# Contributing
 
-We greatly appreciate contributions to MVT!
-
-Your involvement, whether through identifying issues, improving functionality, or enhancing documentation, is very much appreciated. To ensure smooth collaboration and a welcoming environment, we've outlined some key guidelines for contributing below.
-
-## Getting started
-
-Contributing to an open-source project like MVT might seem overwhelming at first, but we're here to support you!
-
- Whether you're a technologist, a frontline human rights defender, a field researcher, or someone new to consensual spyware forensics, there are many ways to make meaningful contributions.
-
- Here's how you can get started:
-
-1. **Explore the codebase:**
-    - Browse the repository to get familar with MVT. Many MVT modules are simple in functionality and easy to understand.
-    - Look for `TODO:` or `FIXME:` comments in the code for areas that need attention.
-
-2. **Check Github issues:**
-    - Look for issues tagged with ["help wanted"](https://github.com/mvt-project/mvt/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22) or ["good first issue"](https://github.com/mvt-project/mvt/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22) to find tasks that are beginner-friendly or where input from the community would be helpful.
-
-3. **Ask for guidance:**
-
-    - If you're unsure where to start, feel free to open a [discussion](https://github.com/mvt-project/mvt/discussions) or comment on an issue.
-
-## How to contribute:
-
-1. **Report issues:**
-
- - Found a bug? Please check existing issues to see if it's already reported. If not, open a new issue. Mobile operating systems and databases are constantly evolving, an new errors may appear spontaniously in new app versions.
-
- **Please provide as much information as possible about the prodblem including: any error messages, steps to reproduce the problem, and any logs or screenshots that can help.**
+Thank you for your interest in contributing to Mobile Verification Toolkit (MVT)! Your help is very much appreciated.
 
 
-2. **Suggest features:**
-    - If you have an idea for new functionality, create a feature request issue and describe your proposal.
+## Where to start
 
-3. **Submit code:**
-    - Fork the repository and create a new branch for your changes.
-    - Ensure your changes align with the code style guidelines (see below).
-    - Open a pull request (PR) with a clear description of your changes and link it to any relevant issues.
+Starting to contribute to a somewhat complex project like MVT might seem intimidating. Unless you have specific ideas of new functionality you would like to submit, some good starting points are searching for `TODO:` and `FIXME:` comments throughout the code. Alternatively you can check if any GitHub issues existed marked with the ["help wanted"](https://github.com/mvt-project/mvt/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22) tag.
 
-4. **Documentation contributions:**
-    - Improving documentation is just as valuable as contributing code! If you notice gaps or inaccuracies in the documentation, feel free to submit changes or suggest updates.
 
 ## Code style
-Please follow these code style guidelines for consistency and readability:
 
-- **Indentation**: use 4 spaces per tab.
-- **Quotes**: Use double quotes (`"`) by default. Use single quotes (`'`) for nested strings instead of escaping (`\"`), or when using f-formatting.
-- **Maximum line length**:
-    - Aim for lines no longer than 80 characters.
-    - Exceptions are allowed for long log lines or strings, which may extend up to 100 characters.
-    - Wrap lines that exceed 100 characters.
+When contributing code to 
 
-Follow [PEP 8 guidelines](https://peps.python.org/pep-0008/) for indentation and overall Python code style. All MVT code is automatically linted with [Ruff](https://github.com/astral-sh/ruff) before merging.
+- **Indentation**: we use 4-spaces tabs.
 
-Please check your code before opening a pull request by running `make ruff`
+- **Quotes**: we use double quotes (`"`) as a default. Single quotes (`'`) can be favored with nested strings instead of escaping (`\"`), or when using f-formatting.
 
-
-## Community and support
-
-We aim to create a supportive and collaborative environment for all contributors. If you run into any challenges, feel free to reach out through the discussions or issues section of the repository.
-
-Your contributions, big or small, help improve MVT and are always appreciated.
+- **Maximum line length**: we strongly encourage to respect a 80 characters long lines and to follow [PEP8 indentation guidelines](https://peps.python.org/pep-0008/#indentation) when having to wrap. However, if breaking at 80 is not possible or is detrimental to the readability of the code, exceptions are tolerated. For example, long log lines, or long strings can be extended to 100 characters long. Please hard wrap anything beyond 100 characters.

Makefile

@@ -27,8 +27,9 @@ test-requirements:
 
 generate-proto-parsers:
 	# Generate python parsers for protobuf files
-	PROTO_FILES=$$(find src/mvt/android/parsers/proto/ -iname "*.proto"); \
-	protoc -Isrc/mvt/android/parsers/proto/ --python_betterproto_out=src/mvt/android/parsers/proto/ $$PROTO_FILES
+	PROTO_DIR="src/mvt/android/parsers/proto/"; \
+	PROTO_FILES=$$(find $(PROTO_DIR) -iname "*.proto"); \
+	protoc -I$(PROTO_DIR) --python_betterproto_out=$(PROTO_DIR) $$PROTO_FILES
 
 clean:
 	rm -rf $(PWD)/build $(PWD)/dist $(PWD)/src/mvt.egg-info

docs/command_completion.md

@@ -1,43 +0,0 @@
-# Command Completion 
-
-MVT utilizes the [Click](https://click.palletsprojects.com/en/stable/) library for creating its command line interface. 
-
-Click provides tab completion support for Bash (version 4.4 and up), Zsh, and Fish.
-
-To enable it, you need to manually register a special function with your shell, which varies depending on the shell you are using.
-
-The following describes how to generate the command completion scripts and add them to your shell configuration. 
-
-> **Note: You will need to start a new shell for the changes to take effect.**
-
-### For Bash
-
-```bash
-# Generates bash completion scripts
-echo "$(_MVT_IOS_COMPLETE=bash_source mvt-ios)" > ~/.mvt-ios-complete.bash &&
-echo "$(_MVT_ANDROID_COMPLETE=bash_source mvt-android)" > ~/.mvt-android-complete.bash
-```
-
-Add the following to `~/.bashrc`:
-```bash
-# source mvt completion scripts
-. ~/.mvt-ios-complete.bash && .  ~/.mvt-android-complete.bash
-```
-
-### For Zsh
-
-```bash
-# Generates zsh completion scripts
-echo "$(_MVT_IOS_COMPLETE=zsh_source mvt-ios)" >  ~/.mvt-ios-complete.zsh &&
-echo "$(_MVT_ANDROID_COMPLETE=zsh_source mvt-android)" > ~/.mvt-android-complete.zsh
-```
-
-Add the following to `~/.zshrc`:
-```bash
-# source mvt completion scripts
-.  ~/.mvt-ios-complete.zsh  && .  ~/.mvt-android-complete.zsh
-```
-
-For more information, visit the official [Click Docs](https://click.palletsprojects.com/en/stable/shell-completion/#enabling-completion).
-
-

docs/install.md

@@ -98,7 +98,3 @@ You now should have the `mvt-ios` and `mvt-android` utilities installed.
 **Notes:**
 1. The `--force` flag is necessary to force the reinstallation of the package.
 2. To revert to using a PyPI version, it will be necessary to `pipx uninstall mvt` first.
-
-## Setting up command completions
-
-See ["Command completions"](command_completion.md)

pyproject.toml

@@ -33,9 +33,6 @@ dependencies = [
     "pyyaml >=6.0",
     "pyahocorasick >= 2.0.0",
     "betterproto >=1.2.0",
-    "pydantic >= 2.10.0",
-    "pydantic-settings >= 2.7.0",
-    'backports.zoneinfo; python_version < "3.9"',
 ]
 requires-python = ">= 3.8"
 

src/mvt/android/artifacts/dumpsys_adb.py

@@ -10,7 +10,7 @@ from .artifact import AndroidArtifact
 
 
 class DumpsysADBArtifact(AndroidArtifact):
-    multiline_fields = ["user_keys", "keystore"]
+    multiline_fields = ["user_keys"]
 
     def indented_dump_parser(self, dump_data):
         """
@@ -67,28 +67,9 @@ class DumpsysADBArtifact(AndroidArtifact):
 
         return res
 
-    def parse_xml(self, xml_data):
-        """
-        Parse XML data from dumpsys ADB output
-        """
-        import xml.etree.ElementTree as ET
-
-        keystore = []
-        keystore_root = ET.fromstring(xml_data)
-        for adb_key in keystore_root.findall("adbKey"):
-            key_info = self.calculate_key_info(adb_key.get("key").encode("utf-8"))
-            key_info["last_connected"] = adb_key.get("lastConnection")
-            keystore.append(key_info)
-
-        return keystore
-
     @staticmethod
     def calculate_key_info(user_key: bytes) -> str:
-        if b" " in user_key:
-            key_base64, user = user_key.split(b" ", 1)
-        else:
-            key_base64, user = user_key, b""
-
+        key_base64, user = user_key.split(b" ", 1)
         key_raw = base64.b64decode(key_base64)
         key_fingerprint = hashlib.md5(key_raw).hexdigest().upper()
         key_fingerprint_colon = ":".join(
@@ -134,24 +115,8 @@ class DumpsysADBArtifact(AndroidArtifact):
         if parsed.get("debugging_manager") is None:
             self.log.error("Unable to find expected ADB entries in dumpsys output")  # noqa
             return
-
-        # Keystore can be in different levels, as the basic parser
-        # is not always consistent due to different dumpsys formats.
-        if parsed.get("keystore"):
-            keystore_data = b"\n".join(parsed["keystore"])
-        elif parsed["debugging_manager"].get("keystore"):
-            keystore_data = b"\n".join(parsed["debugging_manager"]["keystore"])
         else:
-            keystore_data = None
-
-        # Keystore is in XML format on some devices and we need to parse it
-        if keystore_data and keystore_data.startswith(b"<?xml"):
-            parsed["debugging_manager"]["keystore"] = self.parse_xml(keystore_data)
-        else:
-            # Keystore is not XML format
-            parsed["debugging_manager"]["keystore"] = keystore_data
-
-        parsed = parsed["debugging_manager"]
+            parsed = parsed["debugging_manager"]
 
         # Calculate key fingerprints for better readability
         key_info = []

src/mvt/android/artifacts/dumpsys_appops.py

@@ -11,10 +11,6 @@ from mvt.common.utils import convert_datetime_to_iso
 from .artifact import AndroidArtifact
 
 
-RISKY_PERMISSIONS = ["REQUEST_INSTALL_PACKAGES"]
-RISKY_PACKAGES = ["com.android.shell"]
-
-
 class DumpsysAppopsArtifact(AndroidArtifact):
     """
     Parser for dumpsys app ops info
@@ -49,39 +45,15 @@ class DumpsysAppopsArtifact(AndroidArtifact):
                     self.detected.append(result)
                     continue
 
-            detected_permissions = []
             for perm in result["permissions"]:
                 if (
-                    perm["name"] in RISKY_PERMISSIONS
-                    # and perm["access"] == "allow"
+                    perm["name"] == "REQUEST_INSTALL_PACKAGES"
+                    and perm["access"] == "allow"
                 ):
-                    detected_permissions.append(perm)
-                    for entry in sorted(perm["entries"], key=lambda x: x["timestamp"]):
-                        self.log.warning(
-                            "Package '%s' had risky permission '%s' set to '%s' at %s",
-                            result["package_name"],
-                            perm["name"],
-                            entry["access"],
-                            entry["timestamp"],
-                        )
-
-                elif result["package_name"] in RISKY_PACKAGES:
-                    detected_permissions.append(perm)
-                    for entry in sorted(perm["entries"], key=lambda x: x["timestamp"]):
-                        self.log.warning(
-                            "Risky package '%s' had '%s' permission set to '%s' at %s",
-                            result["package_name"],
-                            perm["name"],
-                            entry["access"],
-                            entry["timestamp"],
-                        )
-
-            if detected_permissions:
-                # We clean the result to only include the risky permission, otherwise the timeline
-                # will be polluted with all the other irrelevant permissions
-                cleaned_result = result.copy()
-                cleaned_result["permissions"] = detected_permissions
-                self.detected.append(cleaned_result)
+                    self.log.info(
+                        "Package %s with REQUEST_INSTALL_PACKAGES " "permission",
+                        result["package_name"],
+                    )
 
     def parse(self, output: str) -> None:
         self.results: List[Dict[str, Any]] = []
@@ -149,16 +121,11 @@ class DumpsysAppopsArtifact(AndroidArtifact):
             if line.startswith("          "):
                 # Permission entry like:
                 # Reject: [fg-s]2021-05-19 22:02:52.054 (-314d1h25m2s33ms)
-                access_type = line.split(":")[0].strip()
-                if access_type not in ["Access", "Reject"]:
-                    # Skipping invalid access type. Some entries are not in the format we expect
-                    continue
-
                 if entry:
                     perm["entries"].append(entry)
                     entry = {}
 
-                entry["access"] = access_type
+                entry["access"] = line.split(":")[0].strip()
                 entry["type"] = line[line.find("[") + 1 : line.find("]")]
 
                 try:

src/mvt/android/artifacts/dumpsys_packages.py

@@ -16,7 +16,8 @@ class DumpsysPackagesArtifact(AndroidArtifact):
         for result in self.results:
             if result["package_name"] in ROOT_PACKAGES:
                 self.log.warning(
-                    'Found an installed package related to rooting/jailbreaking: "%s"',
+                    "Found an installed package related to "
+                    'rooting/jailbreaking: "%s"',
                     result["package_name"],
                 )
                 self.detected.append(result)

src/mvt/android/artifacts/dumpsys_platform_compat.py

@@ -1,42 +0,0 @@
-# Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021-2023 The MVT Authors.
-# Use of this software is governed by the MVT License 1.1 that can be found at
-#   https://license.mvt.re/1.1/
-
-from .artifact import AndroidArtifact
-
-
-class DumpsysPlatformCompatArtifact(AndroidArtifact):
-    """
-    Parser for uninstalled apps listed in platform_compat section.
-    """
-
-    def check_indicators(self) -> None:
-        if not self.indicators:
-            return
-
-        for result in self.results:
-            ioc = self.indicators.check_app_id(result["package_name"])
-            if ioc:
-                result["matched_indicator"] = ioc
-                self.detected.append(result)
-                continue
-
-    def parse(self, data: str) -> None:
-        for line in data.splitlines():
-            if not line.startswith("ChangeId(168419799; name=DOWNSCALED;"):
-                continue
-
-            if line.strip() == "":
-                break
-
-            # Look for rawOverrides field
-            if "rawOverrides={" in line:
-                # Extract the content inside the braces for rawOverrides
-                overrides_field = line.split("rawOverrides={", 1)[1].split("};", 1)[0]
-
-                for entry in overrides_field.split(", "):
-                    # Extract app name
-                    uninstall_app = entry.split("=")[0].strip()
-
-                    self.results.append({"package_name": uninstall_app})

src/mvt/android/artifacts/settings.py

@@ -16,11 +16,6 @@ ANDROID_DANGEROUS_SETTINGS = [
         "key": "package_verifier_enable",
         "safe_value": "1",
     },
-    {
-        "description": "disabled APK package verification",
-        "key": "package_verifier_state",
-        "safe_value": "1",
-    },
     {
         "description": "disabled Google Play Protect",
         "key": "package_verifier_user_consent",

src/mvt/android/cmd_check_androidqf.py

@@ -12,6 +12,8 @@ from typing import List, Optional
 from mvt.common.command import Command
 
 from .modules.androidqf import ANDROIDQF_MODULES
+from .modules.bugreport import BUGREPORT_MODULES
+from .modules.bugreport.base import BugReportModule
 
 log = logging.getLogger(__name__)
 
@@ -39,7 +41,11 @@ class CmdAndroidCheckAndroidQF(Command):
         )
 
         self.name = "check-androidqf"
-        self.modules = ANDROIDQF_MODULES
+
+        # We can load AndroidQF and bugreport modules here, as
+        # AndroidQF dump will contain a bugreport.
+        self.modules = ANDROIDQF_MODULES + BUGREPORT_MODULES
+        # TODO: Check how to namespace and deduplicate modules.
 
         self.format: Optional[str] = None
         self.archive: Optional[zipfile.ZipFile] = None
@@ -54,12 +60,44 @@ class CmdAndroidCheckAndroidQF(Command):
                 for fname in subfiles:
                     file_path = os.path.relpath(os.path.join(root, fname), parent_path)
                     self.files.append(file_path)
+
         elif os.path.isfile(self.target_path):
             self.format = "zip"
             self.archive = zipfile.ZipFile(self.target_path)
             self.files = self.archive.namelist()
 
+    def load_bugreport(self):
+        # Refactor this file list loading
+        # First we need to find the bugreport file location
+        bugreport_zip_path = None
+        for file_name in self.files:
+            if file_name.endswith("bugreport.zip"):
+                bugreport_zip_path = file_name
+                break
+        else:
+            self.log.warning("No bugreport.zip found in the AndroidQF dump")
+            return None
+
+        if self.format == "zip":
+            # Create handle to the bugreport.zip file inside the AndroidQF dump
+            handle = self.archive.open(bugreport_zip_path)
+            bugreport_zip = zipfile.ZipFile(handle)
+        else:
+            # Load the bugreport.zip file from the extracted AndroidQF dump on disk.
+            parent_path = Path(self.target_path).absolute().parent.as_posix()
+            bug_report_path = os.path.join(parent_path, bugreport_zip_path)
+            bugreport_zip = zipfile.ZipFile(bug_report_path)
+
+        return bugreport_zip
+
     def module_init(self, module):
+        if isinstance(module, BugReportModule):
+            bugreport_archive = self.load_bugreport()
+            if not bugreport_archive:
+                return
+            module.from_zip(bugreport_archive, bugreport_archive.namelist())
+            return
+
         if self.format == "zip":
             module.from_zip_file(self.archive, self.files)
         else:

src/mvt/android/modules/adb/base.py

@@ -326,7 +326,8 @@ class AndroidExtraction(MVTModule):
 
         if not header["backup"]:
             self.log.error(
-                "Extracting SMS via Android backup failed. No valid backup data found."
+                "Extracting SMS via Android backup failed. "
+                "No valid backup data found."
             )
             return None
 

src/mvt/android/modules/adb/packages.py

@@ -75,7 +75,8 @@ class Packages(AndroidExtraction):
         for result in self.results:
             if result["package_name"] in ROOT_PACKAGES:
                 self.log.warning(
-                    'Found an installed package related to rooting/jailbreaking: "%s"',
+                    "Found an installed package related to "
+                    'rooting/jailbreaking: "%s"',
                     result["package_name"],
                 )
                 self.detected.append(result)

src/mvt/android/modules/adb/sms.py

@@ -70,7 +70,7 @@ class SMS(AndroidExtraction):
             "timestamp": record["isodate"],
             "module": self.__class__.__name__,
             "event": f"sms_{record['direction']}",
-            "data": f'{record.get("address", "unknown source")}: "{body}"',
+            "data": f"{record.get('address', 'unknown source')}: \"{body}\"",
         }
 
     def check_indicators(self) -> None:

src/mvt/android/modules/androidqf/__init__.py

@@ -14,7 +14,6 @@ from .dumpsys_receivers import DumpsysReceivers
 from .dumpsys_adb import DumpsysADBState
 from .getprop import Getprop
 from .packages import Packages
-from .dumpsys_platform_compat import DumpsysPlatformCompat
 from .processes import Processes
 from .settings import Settings
 from .sms import SMS
@@ -30,7 +29,6 @@ ANDROIDQF_MODULES = [
     DumpsysBatteryHistory,
     DumpsysADBState,
     Packages,
-    DumpsysPlatformCompat,
     Processes,
     Getprop,
     Settings,

src/mvt/android/modules/androidqf/base.py

@@ -56,17 +56,11 @@ class AndroidQFModule(MVTModule):
         Android log files to UTC/timezone-aware timestamps.
         """
         get_prop_files = self._get_files_by_pattern("*/getprop.txt")
-        if not get_prop_files:
-            self.log.warning(
-                "Could not find getprop.txt file. "
-                "Some timestamps and timeline data may be incorrect."
-            )
-            return None
+        prop_data = self._get_file_content(get_prop_files[0]).decode("utf-8")
 
         from mvt.android.artifacts.getprop import GetProp
 
         properties_artifact = GetProp()
-        prop_data = self._get_file_content(get_prop_files[0]).decode("utf-8")
         properties_artifact.parse(prop_data)
         timezone = properties_artifact.get_device_timezone()
         if timezone:

src/mvt/android/modules/androidqf/dumpsys_platform_compat.py

@@ -1,44 +0,0 @@
-# Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021-2023 The MVT Authors.
-# Use of this software is governed by the MVT License 1.1 that can be found at
-#   https://license.mvt.re/1.1/
-
-import logging
-from typing import Optional
-
-from mvt.android.artifacts.dumpsys_platform_compat import DumpsysPlatformCompatArtifact
-
-from .base import AndroidQFModule
-
-
-class DumpsysPlatformCompat(DumpsysPlatformCompatArtifact, AndroidQFModule):
-    """This module extracts details on uninstalled apps."""
-
-    def __init__(
-        self,
-        file_path: Optional[str] = None,
-        target_path: Optional[str] = None,
-        results_path: Optional[str] = None,
-        module_options: Optional[dict] = None,
-        log: logging.Logger = logging.getLogger(__name__),
-        results: Optional[list] = None,
-    ) -> None:
-        super().__init__(
-            file_path=file_path,
-            target_path=target_path,
-            results_path=results_path,
-            module_options=module_options,
-            log=log,
-            results=results,
-        )
-
-    def run(self) -> None:
-        dumpsys_file = self._get_files_by_pattern("*/dumpsys.txt")
-        if not dumpsys_file:
-            return
-
-        data = self._get_file_content(dumpsys_file[0]).decode("utf-8", errors="replace")
-        content = self.extract_dumpsys_section(data, "DUMP OF SERVICE platform_compat:")
-        self.parse(content)
-
-        self.log.info("Found %d uninstalled apps", len(self.results))

src/mvt/android/modules/androidqf/files.py

@@ -6,11 +6,7 @@
 import datetime
 import json
 import logging
-
-try:
-    import zoneinfo
-except ImportError:
-    from backports import zoneinfo
+from zoneinfo import ZoneInfo
 from typing import Optional, Union
 
 from mvt.android.modules.androidqf.base import AndroidQFModule
@@ -79,7 +75,7 @@ class Files(AndroidQFModule):
         for result in self.results:
             ioc = self.indicators.check_file_path(result["path"])
             if ioc:
-                result["matched_indicator"] = ioc
+                result["matched_indicator"] == ioc
                 self.detected.append(result)
                 continue
 
@@ -112,10 +108,10 @@ class Files(AndroidQFModule):
 
     def run(self) -> None:
         if timezone := self._get_device_timezone():
-            device_timezone = zoneinfo.ZoneInfo(timezone)
+            device_timezone = ZoneInfo(timezone)
         else:
             self.log.warning("Unable to determine device timezone, using UTC")
-            device_timezone = zoneinfo.ZoneInfo("UTC")
+            device_timezone = ZoneInfo("UTC")
 
         for file in self._get_files_by_pattern("*/files.json"):
             rawdata = self._get_file_content(file).decode("utf-8", errors="ignore")

src/mvt/android/modules/androidqf/packages.py

@@ -44,7 +44,8 @@ class Packages(AndroidQFModule):
         for result in self.results:
             if result["name"] in ROOT_PACKAGES:
                 self.log.warning(
-                    'Found an installed package related to rooting/jailbreaking: "%s"',
+                    "Found an installed package related to "
+                    'rooting/jailbreaking: "%s"',
                     result["name"],
                 )
                 self.detected.append(result)

src/mvt/android/modules/backup/helpers.py

@@ -3,11 +3,10 @@
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 
+import os
 
 from rich.prompt import Prompt
 
-from mvt.common.config import settings
-
 MVT_ANDROID_BACKUP_PASSWORD = "MVT_ANDROID_BACKUP_PASSWORD"
 
 
@@ -17,24 +16,24 @@ def cli_load_android_backup_password(log, backup_password):
 
     Used in MVT CLI command parsers.
     """
-    password_from_env_or_config = settings.ANDROID_BACKUP_PASSWORD
+    password_from_env = os.environ.get(MVT_ANDROID_BACKUP_PASSWORD, None)
     if backup_password:
         log.info(
             "Your password may be visible in the process table because it "
             "was supplied on the command line!"
         )
-        if password_from_env_or_config:
+        if password_from_env:
             log.info(
                 "Ignoring %s environment variable, using --backup-password argument instead",
-                "MVT_ANDROID_BACKUP_PASSWORD",
+                MVT_ANDROID_BACKUP_PASSWORD,
             )
         return backup_password
-    elif password_from_env_or_config:
+    elif password_from_env:
         log.info(
-            "Using backup password from %s environment variable or config file",
-            "MVT_ANDROID_BACKUP_PASSWORD",
+            "Using backup password from %s environment variable",
+            MVT_ANDROID_BACKUP_PASSWORD,
         )
-        return password_from_env_or_config
+        return password_from_env
 
 
 def prompt_or_load_android_backup_password(log, module_options):

src/mvt/android/modules/bugreport/__init__.py

@@ -11,11 +11,9 @@ from .battery_history import BatteryHistory
 from .dbinfo import DBInfo
 from .getprop import Getprop
 from .packages import Packages
-from .platform_compat import PlatformCompat
 from .receivers import Receivers
 from .adb_state import DumpsysADBState
 from .fs_timestamps import BugReportTimestamps
-from .tombstones import Tombstones
 
 BUGREPORT_MODULES = [
     Accessibility,
@@ -26,9 +24,7 @@ BUGREPORT_MODULES = [
     DBInfo,
     Getprop,
     Packages,
-    PlatformCompat,
     Receivers,
     DumpsysADBState,
     BugReportTimestamps,
-    Tombstones,
 ]

src/mvt/android/modules/bugreport/base.py

@@ -2,7 +2,7 @@
 # Copyright (c) 2021-2023 The MVT Authors.
 # See the file 'LICENSE' for usage and copying permissions, or find a copy at
 #   https://github.com/mvt-project/mvt/blob/main/LICENSE
-import datetime
+
 import fnmatch
 import logging
 import os
@@ -92,11 +92,3 @@ class BugReportModule(MVTModule):
                 return None
 
             return self._get_file_content(dumpstate_logs[0])
-
-    def _get_file_modification_time(self, file_path: str) -> dict:
-        if self.zip_archive:
-            file_timetuple = self.zip_archive.getinfo(file_path).date_time
-            return datetime.datetime(*file_timetuple)
-        else:
-            file_stat = os.stat(os.path.join(self.extract_path, file_path))
-            return datetime.datetime.fromtimestamp(file_stat.st_mtime)

src/mvt/android/modules/bugreport/fs_timestamps.py

@@ -3,7 +3,9 @@
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 
+import os
 import logging
+import datetime
 from typing import Optional
 
 from mvt.common.utils import convert_datetime_to_iso
@@ -34,6 +36,14 @@ class BugReportTimestamps(FileTimestampsArtifact, BugReportModule):
             results=results,
         )
 
+    def _get_file_modification_time(self, file_path: str) -> dict:
+        if self.zip_archive:
+            file_timetuple = self.zip_archive.getinfo(file_path).date_time
+            return datetime.datetime(*file_timetuple)
+        else:
+            file_stat = os.stat(os.path.join(self.extract_path, file_path))
+            return datetime.datetime.fromtimestamp(file_stat.st_mtime)
+
     def run(self) -> None:
         filesystem_files = self._get_files_by_pattern("FS/*")
 

src/mvt/android/modules/bugreport/platform_compat.py

@@ -1,48 +0,0 @@
-# Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021-2023 The MVT Authors.
-# Use of this software is governed by the MVT License 1.1 that can be found at
-#   https://license.mvt.re/1.1/
-
-import logging
-from typing import Optional
-
-from mvt.android.artifacts.dumpsys_platform_compat import DumpsysPlatformCompatArtifact
-
-from mvt.android.modules.bugreport.base import BugReportModule
-
-
-class PlatformCompat(DumpsysPlatformCompatArtifact, BugReportModule):
-    """This module extracts details on uninstalled apps."""
-
-    def __init__(
-        self,
-        file_path: Optional[str] = None,
-        target_path: Optional[str] = None,
-        results_path: Optional[str] = None,
-        module_options: Optional[dict] = None,
-        log: logging.Logger = logging.getLogger(__name__),
-        results: Optional[list] = None,
-    ) -> None:
-        super().__init__(
-            file_path=file_path,
-            target_path=target_path,
-            results_path=results_path,
-            module_options=module_options,
-            log=log,
-            results=results,
-        )
-
-    def run(self) -> None:
-        data = self._get_dumpstate_file()
-        if not data:
-            self.log.error(
-                "Unable to find dumpstate file. "
-                "Did you provide a valid bug report archive?"
-            )
-            return
-
-        data = data.decode("utf-8", errors="replace")
-        content = self.extract_dumpsys_section(data, "DUMP OF SERVICE platform_compat:")
-        self.parse(content)
-
-        self.log.info("Found %d uninstalled apps", len(self.results))

src/mvt/android/modules/bugreport/tombstones.py

@@ -42,23 +42,18 @@ class Tombstones(TombstoneCrashArtifact, BugReportModule):
             )
             return
 
-        for tombstone_file in sorted(tombstone_files):
-            tombstone_filename = tombstone_file.split("/")[-1]
-            modification_time = self._get_file_modification_time(tombstone_file)
+        for tombstone_file in tombstone_files:
+            if tombstone_file.endswith("*.pb"):
+                self.log.info("Skipping protobuf tombstone file: %s", tombstone_file)
+                continue
+
+            print(tombstone_file)
             tombstone_data = self._get_file_content(tombstone_file)
+            tombstone = self.parse_tombstone(tombstone_data)
+            print(tombstone)
+            break
 
-            try:
-                if tombstone_file.endswith(".pb"):
-                    self.parse_protobuf(
-                        tombstone_filename, modification_time, tombstone_data
-                    )
-                else:
-                    self.parse(tombstone_filename, modification_time, tombstone_data)
-            except ValueError as e:
-                # Catch any exceptions raised during parsing or validation.
-                self.log.error(f"Error parsing tombstone file {tombstone_file}: {e}")
-
-        self.log.info(
-            "Extracted a total of %d tombstone files",
-            len(self.results),
-        )
+        # self.log.info(
+        #     "Extracted a total of %d database connection pool records",
+        #     len(self.results),
+        # )

src/mvt/common/command.py

@@ -17,7 +17,6 @@ from mvt.common.utils import (
     generate_hashes_from_path,
     get_sha256_from_file_path,
 )
-from mvt.common.config import settings
 from mvt.common.version import MVT_VERSION
 
 
@@ -82,7 +81,7 @@ class Command:
             os.path.join(self.results_path, "command.log")
         )
         formatter = logging.Formatter(
-            "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
+            "%(asctime)s - %(name)s - " "%(levelname)s - %(message)s"
         )
         file_handler.setLevel(logging.DEBUG)
         file_handler.setFormatter(formatter)
@@ -101,25 +100,15 @@ class Command:
         if not self.results_path:
             return
 
-        # We use local timestamps in the timeline on Android as many
-        # logs do not contain timezone information.
-        if type(self).__name__.startswith("CmdAndroid"):
-            is_utc = False
-        else:
-            is_utc = True
-
         if len(self.timeline) > 0:
             save_timeline(
-                self.timeline,
-                os.path.join(self.results_path, "timeline.csv"),
-                is_utc=is_utc,
+                self.timeline, os.path.join(self.results_path, "timeline.csv")
             )
 
         if len(self.timeline_detected) > 0:
             save_timeline(
                 self.timeline_detected,
                 os.path.join(self.results_path, "timeline_detected.csv"),
-                is_utc=is_utc,
             )
 
     def _store_info(self) -> None:
@@ -143,7 +132,7 @@ class Command:
             if ioc_file_path and ioc_file_path not in info["ioc_files"]:
                 info["ioc_files"].append(ioc_file_path)
 
-        if self.target_path and (settings.HASH_FILES or self.hashes):
+        if self.target_path and (os.environ.get("MVT_HASH_FILES") or self.hashes):
             self.generate_hashes()
 
         info["hashes"] = self.hash_values
@@ -152,7 +141,7 @@ class Command:
         with open(info_path, "w+", encoding="utf-8") as handle:
             json.dump(info, handle, indent=4)
 
-        if self.target_path and (settings.HASH_FILES or self.hashes):
+        if self.target_path and (os.environ.get("MVT_HASH_FILES") or self.hashes):
             info_hash = get_sha256_from_file_path(info_path)
             self.log.info('Reference hash of the info.json file: "%s"', info_hash)
 

src/mvt/common/config.py

@@ -1,105 +0,0 @@
-import os
-import yaml
-import json
-
-from typing import Tuple, Type, Optional
-from appdirs import user_config_dir
-from pydantic import AnyHttpUrl, Field
-from pydantic_settings import (
-    BaseSettings,
-    InitSettingsSource,
-    PydanticBaseSettingsSource,
-    SettingsConfigDict,
-    YamlConfigSettingsSource,
-)
-
-MVT_CONFIG_FOLDER = user_config_dir("mvt")
-MVT_CONFIG_PATH = os.path.join(MVT_CONFIG_FOLDER, "config.yaml")
-
-
-class MVTSettings(BaseSettings):
-    model_config = SettingsConfigDict(
-        env_prefix="MVT_",
-        env_nested_delimiter="_",
-        extra="ignore",
-        nested_model_default_partial_updates=True,
-    )
-    # Allow to decided if want to load environment variables
-    load_env: bool = Field(True, exclude=True)
-
-    # General settings
-    PYPI_UPDATE_URL: AnyHttpUrl = Field(
-        "https://pypi.org/pypi/mvt/json",
-        validate_default=False,
-    )
-    NETWORK_ACCESS_ALLOWED: bool = True
-    NETWORK_TIMEOUT: int = 15
-
-    # Command default settings, all can be specified by MVT_ prefixed environment variables too.
-    IOS_BACKUP_PASSWORD: Optional[str] = Field(
-        None, description="Default password to use to decrypt iOS backups"
-    )
-    ANDROID_BACKUP_PASSWORD: Optional[str] = Field(
-        None, description="Default password to use to decrypt Android backups"
-    )
-    STIX2: Optional[str] = Field(
-        None, description="List of directories where STIX2 files are stored"
-    )
-    VT_API_KEY: Optional[str] = Field(
-        None, description="API key to use for VirusTotal lookups"
-    )
-    PROFILE: bool = Field(False, description="Profile the execution of MVT modules")
-    HASH_FILES: bool = Field(False, description="Should MVT hash output files")
-
-    @classmethod
-    def settings_customise_sources(
-        cls,
-        settings_cls: Type[BaseSettings],
-        init_settings: InitSettingsSource,
-        env_settings: PydanticBaseSettingsSource,
-        dotenv_settings: PydanticBaseSettingsSource,
-        file_secret_settings: PydanticBaseSettingsSource,
-    ) -> Tuple[PydanticBaseSettingsSource, ...]:
-        sources = (
-            YamlConfigSettingsSource(settings_cls, MVT_CONFIG_PATH),
-            init_settings,
-        )
-        # Load env variables if enabled
-        if init_settings.init_kwargs.get("load_env", True):
-            sources = (env_settings,) + sources
-        return sources
-
-    def save_settings(
-        self,
-    ) -> None:
-        """
-        Save the current settings to a file.
-        """
-        if not os.path.isdir(MVT_CONFIG_FOLDER):
-            os.makedirs(MVT_CONFIG_FOLDER)
-
-        # Dump the settings to the YAML file
-        model_serializable = json.loads(self.model_dump_json(exclude_defaults=True))
-        with open(MVT_CONFIG_PATH, "w") as config_file:
-            config_file.write(yaml.dump(model_serializable, default_flow_style=False))
-
-    @classmethod
-    def initialise(cls) -> "MVTSettings":
-        """
-        Initialise the settings file.
-
-        We first initialise the settings (without env variable) and then persist
-        them to file. This way we can update the config file with the default values.
-
-        Afterwards we load the settings again, this time including the env variables.
-        """
-        # Set invalid env prefix to avoid loading env variables.
-        settings = MVTSettings(load_env=False)
-        settings.save_settings()
-
-        # Load the settings again with any ENV variables.
-        settings = MVTSettings(load_env=True)
-        return settings
-
-
-settings = MVTSettings.initialise()

src/mvt/common/indicators.py

@@ -14,7 +14,6 @@ import ahocorasick
 from appdirs import user_data_dir
 
 from .url import URL
-from .config import settings
 
 MVT_DATA_FOLDER = user_data_dir("mvt")
 MVT_INDICATORS_FOLDER = os.path.join(MVT_DATA_FOLDER, "indicators")
@@ -42,12 +41,12 @@ class Indicators:
 
     def _check_stix2_env_variable(self) -> None:
         """
-        Checks if MVT_STIX2 setting or environment variable contains path to a STIX file. Also recursively searches through dirs in MVT_STIX2
+        Checks if a variable MVT_STIX2 contains path to a STIX file. Also recursively searches through dirs in MVT_STIX2
         """
-        if not settings.STIX2:
+        if "MVT_STIX2" not in os.environ:
             return
 
-        paths = settings.STIX2.split(":")
+        paths = os.environ["MVT_STIX2"].split(":")
         for path in paths:
             if os.path.isfile(path) and path.lower().endswith(".stix2"):
                 self.parse_stix2(path)
@@ -384,7 +383,8 @@ class Indicators:
         for ioc in self.get_iocs("urls"):
             if ioc["value"] == url:
                 self.log.warning(
-                    'Found a known suspicious URL %s matching indicator "%s" from "%s"',
+                    "Found a known suspicious URL %s "
+                    'matching indicator "%s" from "%s"',
                     url,
                     ioc["value"],
                     ioc["name"],

src/mvt/common/logo.py

@@ -29,7 +29,7 @@ def check_updates() -> None:
         if latest_version:
             rich_print(
                 f"\t\t[bold]Version {latest_version} is available! "
-                "Upgrade mvt with `pip3 install -U mvt` or with `pipx upgrade mvt`[/bold]"
+                "Upgrade mvt with `pip3 install -U mvt`[/bold]"
             )
 
     # Then we check for indicators files updates.

src/mvt/common/module.py

@@ -227,7 +227,7 @@ def run_module(module: MVTModule) -> None:
         module.save_to_json()
 
 
-def save_timeline(timeline: list, timeline_path: str, is_utc: bool = True) -> None:
+def save_timeline(timeline: list, timeline_path: str) -> None:
     """Save the timeline in a csv file.
 
     :param timeline: List of records to order and store
@@ -238,12 +238,7 @@ def save_timeline(timeline: list, timeline_path: str, is_utc: bool = True) -> No
         csvoutput = csv.writer(
             handle, delimiter=",", quotechar='"', quoting=csv.QUOTE_ALL, escapechar="\\"
         )
-
-        if is_utc:
-            timestamp_header = "UTC Timestamp"
-        else:
-            timestamp_header = "Device Local Timestamp"
-        csvoutput.writerow([timestamp_header, "Plugin", "Event", "Description"])
+        csvoutput.writerow(["UTC Timestamp", "Plugin", "Event", "Description"])
 
         for event in sorted(
             timeline, key=lambda x: x["timestamp"] if x["timestamp"] is not None else ""