Sorting imports and removing unused ones

Added process to automatically check for indicators updates

.github/workflows/flake8.yml

@@ -0,0 +1,26 @@
+name: Flake8
+
+on:
+  push:
+    paths:
+      - '*.py'
+
+jobs:
+  flake8_py3:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Setup Python
+        uses: actions/setup-python@v1
+        with:
+          python-version: 3.9
+          architecture: x64
+      - name: Checkout
+        uses: actions/checkout@master
+      - name: Install flake8
+        run: pip install flake8
+      - name: Run flake8
+        uses: suo/flake8-github-action@releases/v1
+        with:
+          checkName: 'flake8_py3'   # NOTE: this needs to be the same as the job name
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/python-package.yml

@@ -28,7 +28,7 @@ jobs:
     - name: Install dependencies
       run: |
         python -m pip install --upgrade pip
-        python -m pip install flake8 pytest safety stix2
+        python -m pip install flake8 pytest safety stix2 pytest-mock
         if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
         python -m pip install .
     - name: Lint with flake8

.gitignore

@@ -133,4 +133,7 @@ dmypy.json
 *~
 
 # IDEA Dev Environment
-.idea
+.idea
+
+# Sublime Text project files
+*.sublime*

AUTHORS

@@ -1,7 +0,0 @@
-MVT was originally authored by Claudio Guarnieri <nex@nex.sx>.
-
-For an up-to-date list of all contributors visit:
-    https://github.com/mvt-project/mvt/graphs/contributors
-
-Or run:
-    git shortlog -s -n

README.md

@@ -1,5 +1,5 @@
 <p align="center">
-     <img src="./docs/mvt.png" width="200" />
+     <img src="https://docs.mvt.re/en/latest/mvt.png" width="200" />
 </p>
 
 # Mobile Verification Toolkit
@@ -15,6 +15,7 @@ It has been developed and released by the [Amnesty International Security Lab](h
 
 *Warning*: MVT is a forensic research tool intended for technologists and investigators. Using it requires understanding the basics of forensic analysis and using command-line tools. This is not intended for end-user self-assessment. If you are concerned with the security of your device please seek expert assistance.
 
+
 ## Installation
 
 MVT can be installed from sources or from [PyPi](https://pypi.org/project/mvt/) (you will need some dependencies, check the [documentation](https://docs.mvt.re/en/latest/install/)):
@@ -23,14 +24,14 @@ MVT can be installed from sources or from [PyPi](https://pypi.org/project/mvt/)
 pip3 install mvt
 ```
 
-Alternatively, you can decide to run MVT and all relevant tools through a [Docker container](https://docs.mvt.re/en/latest/docker/).
+For alternative installation options and known issues, please refer to the [documentation](https://docs.mvt.re/en/latest/install/) as well as [GitHub Issues](https://github.com/mvt-project/mvt/issues).
 
-**Please note:** MVT is best run on Linux or Mac systems. [It does not currently support running natively on Windows.](https://docs.mvt.re/en/latest/install/#mvt-on-windows)
 
 ## Usage
 
 MVT provides two commands `mvt-ios` and `mvt-android`. [Check out the documentation to learn how to use them!](https://docs.mvt.re/)
 
+
 ## License
 
 The purpose of MVT is to facilitate the ***consensual forensic analysis*** of devices of those who might be targets of sophisticated mobile spyware attacks, especially members of civil society and marginalized communities. We do not want MVT to enable privacy violations of non-consenting individuals.  In order to achieve this, MVT is released under its own license. [Read more here.](https://docs.mvt.re/en/latest/license/)

dev/mvt-android

@@ -1,6 +1,6 @@
 #!/usr/bin/env python3
 # Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021-2022 The MVT Project Authors.
+# Copyright (c) 2021-2022 Claudio Guarnieri.
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 

dev/mvt-ios

@@ -1,6 +1,6 @@
 #!/usr/bin/env python3
 # Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021-2022 The MVT Project Authors.
+# Copyright (c) 2021-2022 Claudio Guarnieri.
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 

docs/android/backup.md

@@ -52,4 +52,4 @@ If the backup is encrypted, ABE will prompt you to enter the password.
 
 Alternatively, [ab-decrypt](https://github.com/joernheissler/ab-decrypt) can be used for that purpose.
 
-You can then extract SMSs containing links with MVT by passing the folder path as parameter instead of the `.ab` file: `mvt-android check-backup --output /path/to/results/ /path/to/backup/`.
+You can then extract SMSs containing links with MVT by passing the folder path as parameter instead of the `.ab` file: `mvt-android check-backup --output /path/to/results/ /path/to/backup/` (the path to backup given should be the folder containing the `apps` folder).

docs/android/download_apks.md

@@ -13,22 +13,16 @@ It might take several minutes to complete.
 !!! info
     MVT will likely warn you it was unable to download certain installed packages. There is no reason to be alarmed: this is typically expected behavior when MVT attempts to download a system package it has no privileges to access.
 
-Optionally, you can decide to enable lookups of the SHA256 hash of all the extracted APKs on [VirusTotal](https://www.virustotal.com) and/or [Koodous](https://koodous.com). While these lookups do not provide any conclusive assessment on all of the extracted APKs, they might highlight any known malicious ones:
+Optionally, you can decide to enable lookups of the SHA256 hash of all the extracted APKs on [VirusTotal](https://www.virustotal.com). While these lookups do not provide any conclusive assessment on all of the extracted APKs, they might highlight any known malicious ones:
 
 ```bash
-mvt-android download-apks --output /path/to/folder --virustotal
-mvt-android download-apks --output /path/to/folder --koodous
+MVT_VT_API_KEY=<key> mvt-android download-apks --output /path/to/folder --virustotal
 ```
 
-Or, to launch all available lookups:
+Please note that in order to use VirusTotal lookups you are required to provide your own API key through the `MVT_VT_API_KEY` environment variable. You should also note that VirusTotal enforces strict API usage. Be mindful that MVT might consume your hourly search quota.
+
+In case you have a previous extraction of APKs you want to later check against VirusTotal, you can do so with the following arguments:
 
 ```bash
-mvt-android download-apks --output /path/to/folder --all-checks
+MVT_VT_API_KEY=<key> mvt-android download-apks --from-file /path/to/folder/apks.json --virustotal
 ```
-
-In case you have a previous extraction of APKs you want to later check against VirusTotal and Koodous, you can do so with the following arguments:
-
-```bash
-mvt-android download-apks --from-file /path/to/folder/apks.json --all-checks
-```
-

docs/android/methodology.md

@@ -8,8 +8,10 @@ However, not all is lost.
 
 Because malware attacks over Android typically take the form of malicious or backdoored apps, the very first thing you might want to do is to extract and verify all installed Android packages and triage quickly if there are any which stand out as malicious or which might be atypical.
 
-While it is out of the scope of this documentation to dwell into details on how to analyze Android apps, MVT does allow to easily and automatically extract information about installed apps, download copies of them, and quickly lookup services such as [VirusTotal](https://www.virustotal.com) or [Koodous](https://koodous.com) which might quickly indicate known bad apps.
+While it is out of the scope of this documentation to dwell into details on how to analyze Android apps, MVT does allow to easily and automatically extract information about installed apps, download copies of them, and quickly look them up on services such as [VirusTotal](https://www.virustotal.com).
 
+!!! info "Using VirusTotal"
+	Please note that in order to use VirusTotal lookups you are required to provide your own API key through the `MVT_VT_API_KEY` environment variable. You should also note that VirusTotal enforces strict API usage. Be mindful that MVT might consume your hourly search quota.
 
 ## Check the device over Android Debug Bridge
 

docs/iocs.md

@@ -39,8 +39,8 @@ export MVT_STIX2="/home/user/IOC1.stix2:/home/user/IOC2.stix2"
 - The [Amnesty International investigations repository](https://github.com/AmnestyTech/investigations) contains STIX-formatted IOCs for:
     - [Pegasus](https://en.wikipedia.org/wiki/Pegasus_(spyware)) ([STIX2](https://raw.githubusercontent.com/AmnestyTech/investigations/master/2021-07-18_nso/pegasus.stix2))
     - [Predator from Cytrox](https://citizenlab.ca/2021/12/pegasus-vs-predator-dissidents-doubly-infected-iphone-reveals-cytrox-mercenary-spyware/) ([STIX2](https://raw.githubusercontent.com/AmnestyTech/investigations/master/2021-12-16_cytrox/cytrox.stix2))
-- [This repository](https://github.com/Te-k/stalkerware-indicators) contains IOCs for Android stalkerware including [a STIX MVT-compatible file](https://raw.githubusercontent.com/Te-k/stalkerware-indicators/master/stalkerware.stix2).
+- [This repository](https://github.com/Te-k/stalkerware-indicators) contains IOCs for Android stalkerware including [a STIX MVT-compatible file](https://raw.githubusercontent.com/Te-k/stalkerware-indicators/master/generated/stalkerware.stix2).
 
-You can automaticallly download the latest public indicator files with the command `mvt-ios download-iocs` or `mvt-android download-iocs`. These commands download the list of indicators listed [here](https://github.com/mvt-project/mvt/blob/main/public_indicators.json) and store them in the [appdir](https://pypi.org/project/appdirs/) folder. They are then loaded automatically by mvt.
+You can automaticallly download the latest public indicator files with the command `mvt-ios download-iocs` or `mvt-android download-iocs`. These commands download the list of indicators listed [here](https://github.com/mvt-project/mvt/blob/main/public_indicators.json) and store them in the [appdir](https://pypi.org/project/appdirs/) folder. They are then loaded automatically by MVT.
 
 Please [open an issue](https://github.com/mvt-project/mvt/issues/) to suggest new sources of STIX-formatted IOCs.

docs/ios/records.md

@@ -18,7 +18,7 @@ If indicators are provided through the command-line, processes and domains are c
 
 ### `backup_info.json`
 
-!!! info "Availabiliy"
+!!! info "Availability"
     Backup: :material-check:
     Full filesystem dump: :material-close:
 

mkdocs.yml

@@ -1,7 +1,7 @@
 site_name: Mobile Verification Toolkit
 repo_url: https://github.com/mvt-project/mvt
 edit_uri: edit/main/docs/
-copyright: Copyright © 2021 MVT Project Developers
+copyright: Copyright © 2021-2022 MVT Project Developers
 site_description: Mobile Verification Toolkit Documentation
 markdown_extensions:
     - attr_list

mvt/__init__.py

@@ -1,4 +1,4 @@
 # Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021-2022 The MVT Project Authors.
+# Copyright (c) 2021-2022 Claudio Guarnieri.
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/

mvt/android/__init__.py

@@ -1,5 +1,5 @@
 # Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021-2022 The MVT Project Authors.
+# Copyright (c) 2021-2022 Claudio Guarnieri.
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 

mvt/android/cmd_check_adb.py

@@ -0,0 +1,25 @@
+# Mobile Verification Toolkit (MVT)
+# Copyright (c) 2021-2022 Claudio Guarnieri.
+# Use of this software is governed by the MVT License 1.1 that can be found at
+#   https://license.mvt.re/1.1/
+
+import logging
+
+from mvt.common.command import Command
+
+from .modules.adb import ADB_MODULES
+
+log = logging.getLogger(__name__)
+
+
+class CmdAndroidCheckADB(Command):
+
+    name = "check-adb"
+    modules = ADB_MODULES
+
+    def __init__(self, target_path: str = None, results_path: str = None,
+                 ioc_files: list = [], module_name: str = None, serial: str = None,
+                 fast_mode: bool = False):
+        super().__init__(target_path=target_path, results_path=results_path,
+                         ioc_files=ioc_files, module_name=module_name,
+                         serial=serial, fast_mode=fast_mode, log=log)

mvt/android/cmd_check_backup.py

@@ -0,0 +1,83 @@
+# Mobile Verification Toolkit (MVT)
+# Copyright (c) 2021-2022 Claudio Guarnieri.
+# Use of this software is governed by the MVT License 1.1 that can be found at
+#   https://license.mvt.re/1.1/
+
+import getpass
+import io
+import logging
+import os
+import sys
+import tarfile
+from pathlib import Path
+
+from mvt.android.parsers.backup import (AndroidBackupParsingError,
+                                        InvalidBackupPassword, parse_ab_header,
+                                        parse_backup_file)
+from mvt.common.command import Command
+
+from .modules.backup import BACKUP_MODULES
+
+log = logging.getLogger(__name__)
+
+
+class CmdAndroidCheckBackup(Command):
+
+    name = "check-backup"
+    modules = BACKUP_MODULES
+
+    def __init__(self, target_path: str = None, results_path: str = None,
+                 ioc_files: list = [], module_name: str = None, serial: str = None,
+                 fast_mode: bool = False):
+        super().__init__(target_path=target_path, results_path=results_path,
+                         ioc_files=ioc_files, module_name=module_name,
+                         serial=serial, fast_mode=fast_mode, log=log)
+
+        self.backup_type = None
+        self.backup_archive = None
+        self.backup_files = []
+
+    def init(self):
+        if os.path.isfile(self.target_path):
+            self.backup_type = "ab"
+            with open(self.target_path, "rb") as handle:
+                data = handle.read()
+
+            header = parse_ab_header(data)
+            if not header["backup"]:
+                log.critical("Invalid backup format, file should be in .ab format")
+                sys.exit(1)
+
+            password = None
+            if header["encryption"] != "none":
+                password = getpass.getpass(prompt="Backup Password: ", stream=None)
+            try:
+                tardata = parse_backup_file(data, password=password)
+            except InvalidBackupPassword:
+                log.critical("Invalid backup password")
+                sys.exit(1)
+            except AndroidBackupParsingError as e:
+                log.critical("Impossible to parse this backup file: %s", e)
+                log.critical("Please use Android Backup Extractor (ABE) instead")
+                sys.exit(1)
+
+            dbytes = io.BytesIO(tardata)
+            self.backup_archive = tarfile.open(fileobj=dbytes)
+            for member in self.backup_archive:
+                self.backup_files.append(member.name)
+
+        elif os.path.isdir(self.target_path):
+            self.backup_type = "folder"
+            self.target_path = Path(self.target_path).absolute().as_posix()
+            for root, subdirs, subfiles in os.walk(os.path.abspath(self.target_path)):
+                for fname in subfiles:
+                    self.backup_files.append(os.path.relpath(os.path.join(root, fname), self.target_path))
+        else:
+            log.critical("Invalid backup path, path should be a folder or an Android Backup (.ab) file")
+            sys.exit(1)
+
+    def module_init(self, module):
+        if self.backup_type == "folder":
+            module.from_folder(self.target_path, self.backup_files)
+        else:
+            module.from_ab(self.target_path, self.backup_archive, self.backup_files)

mvt/android/cmd_check_bugreport.py

@@ -0,0 +1,51 @@
+# Mobile Verification Toolkit (MVT)
+# Copyright (c) 2021-2022 Claudio Guarnieri.
+# Use of this software is governed by the MVT License 1.1 that can be found at
+#   https://license.mvt.re/1.1/
+
+import logging
+import os
+from pathlib import Path
+from zipfile import ZipFile
+
+from mvt.common.command import Command
+
+from .modules.bugreport import BUGREPORT_MODULES
+
+log = logging.getLogger(__name__)
+
+
+class CmdAndroidCheckBugreport(Command):
+
+    name = "check-bugreport"
+    modules = BUGREPORT_MODULES
+
+    def __init__(self, target_path: str = None, results_path: str = None,
+                 ioc_files: list = [], module_name: str = None, serial: str = None,
+                 fast_mode: bool = False):
+        super().__init__(target_path=target_path, results_path=results_path,
+                         ioc_files=ioc_files, module_name=module_name,
+                         serial=serial, fast_mode=fast_mode, log=log)
+
+        self.bugreport_format = None
+        self.bugreport_archive = None
+        self.bugreport_files = []
+
+    def init(self):
+        if os.path.isfile(self.target_path):
+            self.bugreport_format = "zip"
+            self.bugreport_archive = ZipFile(self.target_path)
+            for file_name in self.bugreport_archive.namelist():
+                self.bugreport_files.append(file_name)
+        elif os.path.isdir(self.target_path):
+            self.bugreport_format = "dir"
+            parent_path = Path(self.target_path).absolute().as_posix()
+            for root, subdirs, subfiles in os.walk(os.path.abspath(self.target_path)):
+                for file_name in subfiles:
+                    self.bugreport_files.append(os.path.relpath(os.path.join(root, file_name), parent_path))
+
+    def module_init(self, module):
+        if self.bugreport_format == "zip":
+            module.from_zip(self.bugreport_archive, self.bugreport_files)
+        else:
+            module.from_folder(self.target_path, self.bugreport_files)

mvt/android/cmd_download_apks.py

@@ -1,5 +1,5 @@
 # Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021-2022 The MVT Project Authors.
+# Copyright (c) 2021-2022 Claudio Guarnieri.
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 
@@ -176,7 +176,7 @@ class DownloadAPKs(AndroidExtraction):
         with open(json_path, "w", encoding="utf-8") as handle:
             json.dump(self.packages, handle, indent=4)
 
-    def run(self):
+    def run(self) -> None:
         """Run all steps of fetch-apk."""
         self.get_packages()
         self._adb_connect()

mvt/android/data/root_binaries.txt

@@ -1,10 +0,0 @@
-su
-busybox
-supersu
-Superuser.apk
-KingoUser.apk
-SuperSu.apk
-magisk
-magiskhide
-magiskinit
-magiskpolicy

mvt/android/data/root_packages.txt

@@ -1,25 +0,0 @@
-com.noshufou.android.su
-com.noshufou.android.su.elite
-eu.chainfire.supersu
-com.koushikdutta.superuser
-com.thirdparty.superuser
-com.yellowes.su
-com.koushikdutta.rommanager
-com.koushikdutta.rommanager.license
-com.dimonvideo.luckypatcher
-com.chelpus.lackypatch
-com.ramdroid.appquarantine
-com.ramdroid.appquarantinepro
-com.devadvance.rootcloak
-com.devadvance.rootcloakplus
-de.robv.android.xposed.installer
-com.saurik.substrate
-com.zachspong.temprootremovejb
-com.amphoras.hidemyroot
-com.amphoras.hidemyrootadfree
-com.formyhm.hiderootPremium
-com.formyhm.hideroot
-me.phh.superuser
-eu.chainfire.supersu.pro
-com.kingouser.com
-com.topjohnwu.magisk

mvt/android/lookups/__init__.py

@@ -1,4 +0,0 @@
-# Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021-2022 The MVT Project Authors.
-# Use of this software is governed by the MVT License 1.1 that can be found at
-#   https://license.mvt.re/1.1/

mvt/android/lookups/koodous.py

@@ -1,58 +0,0 @@
-# Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021-2022 The MVT Project Authors.
-# Use of this software is governed by the MVT License 1.1 that can be found at
-#   https://license.mvt.re/1.1/
-
-import logging
-
-import requests
-from rich.console import Console
-from rich.progress import track
-from rich.table import Table
-from rich.text import Text
-
-log = logging.getLogger(__name__)
-
-
-def koodous_lookup(packages):
-    log.info("Looking up all extracted files on Koodous (www.koodous.com)")
-    log.info("This might take a while...")
-
-    table = Table(title="Koodous Packages Detections")
-    table.add_column("Package name")
-    table.add_column("File name")
-    table.add_column("Trusted")
-    table.add_column("Detected")
-    table.add_column("Rating")
-
-    total_packages = len(packages)
-    for i in track(range(total_packages), description=f"Looking up {total_packages} packages..."):
-        package = packages[i]
-        for file in package.get("files", []):
-            url = f"https://api.koodous.com/apks/{file['sha256']}"
-            res = requests.get(url)
-            report = res.json()
-
-            row = [package["package_name"], file["path"]]
-
-            if "package_name" in report:
-                trusted = "no"
-                if report["trusted"]:
-                    trusted = Text("yes", "green bold")
-
-                detected = "no"
-                if report["detected"]:
-                    detected = Text("yes", "red bold")
-
-                rating = "0"
-                if int(report["rating"]) < 0:
-                    rating = Text(str(report["rating"]), "red bold")
-
-                row.extend([trusted, detected, rating])
-            else:
-                row.extend(["n/a", "n/a", "n/a"])
-
-            table.add_row(*row)
-
-    console = Console()
-    console.print(table)

mvt/android/lookups/virustotal.py

@@ -1,100 +0,0 @@
-# Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021-2022 The MVT Project Authors.
-# Use of this software is governed by the MVT License 1.1 that can be found at
-#   https://license.mvt.re/1.1/
-
-import logging
-
-import requests
-from rich.console import Console
-from rich.progress import track
-from rich.table import Table
-from rich.text import Text
-
-log = logging.getLogger(__name__)
-
-
-def get_virustotal_report(hashes):
-    apikey = "233f22e200ca5822bd91103043ccac138b910db79f29af5616a9afe8b6f215ad"
-    url = f"https://www.virustotal.com/partners/sysinternals/file-reports?apikey={apikey}"
-
-    items = []
-    for sha256 in hashes:
-        items.append({
-            "autostart_location": "",
-            "autostart_entry": "",
-            "hash": sha256,
-            "local_name": "",
-            "creation_datetime": "",
-        })
-    headers = {"User-Agent": "VirusTotal", "Content-Type": "application/json"}
-    res = requests.post(url, headers=headers, json=items)
-
-    if res.status_code == 200:
-        report = res.json()
-        return report["data"]
-    else:
-        log.error("Unexpected response from VirusTotal: %s", res.status_code)
-        return None
-
-
-def virustotal_lookup(packages):
-    # NOTE: This is temporary, until we resolved the issue.
-    log.error("Unfortunately VirusTotal lookup is disabled until further notice, due to unresolved issues with the API service.")
-    return
-
-    log.info("Looking up all extracted files on VirusTotal (www.virustotal.com)")
-
-    unique_hashes = []
-    for package in packages:
-        for file in package.get("files", []):
-            if file["sha256"] not in unique_hashes:
-                unique_hashes.append(file["sha256"])
-
-    total_unique_hashes = len(unique_hashes)
-
-    detections = {}
-
-    def virustotal_query(batch):
-        report = get_virustotal_report(batch)
-        if not report:
-            return
-
-        for entry in report:
-            if entry["hash"] not in detections and entry["found"] is True:
-                detections[entry["hash"]] = entry["detection_ratio"]
-
-    batch = []
-    for i in track(range(total_unique_hashes), description=f"Looking up {total_unique_hashes} files..."):
-        file_hash = unique_hashes[i]
-        batch.append(file_hash)
-        if len(batch) == 25:
-            virustotal_query(batch)
-            batch = []
-
-    if batch:
-        virustotal_query(batch)
-
-    table = Table(title="VirusTotal Packages Detections")
-    table.add_column("Package name")
-    table.add_column("File path")
-    table.add_column("Detections")
-
-    for package in packages:
-        for file in package.get("files", []):
-            row = [package["package_name"], file["path"]]
-
-            if file["sha256"] in detections:
-                detection = detections[file["sha256"]]
-                positives = detection.split("/")[0]
-                if int(positives) > 0:
-                    row.append(Text(detection, "red bold"))
-                else:
-                    row.append(detection)
-            else:
-                row.append("not found")
-
-            table.add_row(*row)
-
-    console = Console()
-    console.print(table)

mvt/android/modules/__init__.py

@@ -1,4 +1,4 @@
 # Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021-2022 The MVT Project Authors.
+# Copyright (c) 2021-2022 Claudio Guarnieri.
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/

mvt/android/modules/adb/__init__.py

@@ -1,11 +1,12 @@
 # Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021-2022 The MVT Project Authors.
+# Copyright (c) 2021-2022 Claudio Guarnieri.
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 
 from .chrome_history import ChromeHistory
 from .dumpsys_accessibility import DumpsysAccessibility
 from .dumpsys_activities import DumpsysActivities
+from .dumpsys_appops import DumpsysAppOps
 from .dumpsys_battery_daily import DumpsysBatteryDaily
 from .dumpsys_battery_history import DumpsysBatteryHistory
 from .dumpsys_dbinfo import DumpsysDBInfo
@@ -25,5 +26,5 @@ from .whatsapp import Whatsapp
 ADB_MODULES = [ChromeHistory, SMS, Whatsapp, Processes, Getprop, Settings,
                SELinuxStatus, DumpsysBatteryHistory, DumpsysBatteryDaily,
                DumpsysReceivers, DumpsysActivities, DumpsysAccessibility,
-               DumpsysDBInfo, DumpsysFull, Packages, Logcat, RootBinaries,
-               Files]
+               DumpsysDBInfo, DumpsysFull, DumpsysAppOps, Packages, Logcat,
+               RootBinaries, Files]

mvt/android/modules/adb/base.py

@@ -1,5 +1,5 @@
 # Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021-2022 The MVT Project Authors.
+# Copyright (c) 2021-2022 Claudio Guarnieri.
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 
@@ -12,6 +12,7 @@ import string
 import sys
 import tempfile
 import time
+from typing import Callable
 
 from adb_shell.adb_device import AdbDeviceTcp, AdbDeviceUsb
 from adb_shell.auth.keygen import keygen, write_public_keyfile
@@ -33,17 +34,18 @@ ADB_PUB_KEY_PATH = os.path.expanduser("~/.android/adbkey.pub")
 class AndroidExtraction(MVTModule):
     """This class provides a base for all Android extraction modules."""
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
         self.device = None
         self.serial = None
 
     @staticmethod
-    def _adb_check_keys():
+    def _adb_check_keys() -> None:
         """Make sure Android adb keys exist."""
         if not os.path.isdir(os.path.dirname(ADB_KEY_PATH)):
             os.makedirs(os.path.dirname(ADB_KEY_PATH))
@@ -54,7 +56,7 @@ class AndroidExtraction(MVTModule):
         if not os.path.exists(ADB_PUB_KEY_PATH):
             write_public_keyfile(ADB_KEY_PATH, ADB_PUB_KEY_PATH)
 
-    def _adb_connect(self):
+    def _adb_connect(self) -> None:
         """Connect to the device over adb."""
         self._adb_check_keys()
 
@@ -103,17 +105,17 @@ class AndroidExtraction(MVTModule):
             else:
                 break
 
-    def _adb_disconnect(self):
+    def _adb_disconnect(self) -> None:
         """Close adb connection to the device."""
         self.device.close()
 
-    def _adb_reconnect(self):
+    def _adb_reconnect(self) -> None:
         """Reconnect to device using adb."""
         log.info("Reconnecting ...")
         self._adb_disconnect()
         self._adb_connect()
 
-    def _adb_command(self, command):
+    def _adb_command(self, command: str) -> str:
         """Execute an adb shell command.
 
         :param command: Shell command to execute
@@ -122,7 +124,7 @@ class AndroidExtraction(MVTModule):
         """
         return self.device.shell(command, read_timeout_s=200.0)
 
-    def _adb_check_if_root(self):
+    def _adb_check_if_root(self) -> bool:
         """Check if we have a `su` binary on the Android device.
 
 
@@ -131,7 +133,7 @@ class AndroidExtraction(MVTModule):
         """
         return bool(self._adb_command("command -v su"))
 
-    def _adb_root_or_die(self):
+    def _adb_root_or_die(self) -> None:
         """Check if we have a `su` binary, otherwise raise an Exception."""
         if not self._adb_check_if_root():
             raise InsufficientPrivileges("This module is optionally available in case the device is already rooted. Do NOT root your own device!")
@@ -145,7 +147,7 @@ class AndroidExtraction(MVTModule):
         """
         return self._adb_command(f"su -c {command}")
 
-    def _adb_check_file_exists(self, file):
+    def _adb_check_file_exists(self, file: str) -> bool:
         """Verify that a file exists.
 
         :param file: Path of the file
@@ -162,7 +164,9 @@ class AndroidExtraction(MVTModule):
 
         return bool(self._adb_command_as_root(f"[ ! -f {file} ] || echo 1"))
 
-    def _adb_download(self, remote_path, local_path, progress_callback=None, retry_root=True):
+    def _adb_download(self, remote_path: str, local_path: str,
+                      progress_callback: Callable = None,
+                      retry_root: bool = True) -> None:
         """Download a file form the device.
 
         :param remote_path: Path to download from the device
@@ -179,7 +183,8 @@ class AndroidExtraction(MVTModule):
             else:
                 raise Exception(f"Unable to download file {remote_path}: {e}")
 
-    def _adb_download_root(self, remote_path, local_path, progress_callback=None):
+    def _adb_download_root(self, remote_path: str, local_path: str,
+                           progress_callback: Callable = None) -> None:
         try:
             # Check if we have root, if not raise an Exception.
             self._adb_root_or_die()
@@ -207,7 +212,8 @@ class AndroidExtraction(MVTModule):
         except AdbCommandFailureException as e:
             raise Exception(f"Unable to download file {remote_path}: {e}")
 
-    def _adb_process_file(self, remote_path, process_routine):
+    def _adb_process_file(self, remote_path: str,
+                          process_routine: Callable) -> None:
         """Download a local copy of a file which is only accessible as root.
         This is a wrapper around process_routine.
 
@@ -247,13 +253,12 @@ class AndroidExtraction(MVTModule):
         # Disconnect from the device.
         self._adb_disconnect()
 
-    def _generate_backup(self, package_name):
-        # Run ADB command to create a backup of SMS app
+    def _generate_backup(self, package_name: str) -> bytes:
         self.log.warning("Please check phone and accept Android backup prompt. You may need to set a backup password. \a")
 
-        # Run ADB command to create a backup of SMS app
         # TODO: Base64 encoding as temporary fix to avoid byte-mangling over the shell transport...
-        backup_output_b64 = self._adb_command("/system/bin/bu backup -nocompress '{}' | base64".format(package_name))
+        backup_output_b64 = self._adb_command("/system/bin/bu backup -nocompress '{}' | base64".format(
+            package_name))
         backup_output = base64.b64decode(backup_output_b64)
         header = parse_ab_header(backup_output)
 
@@ -264,17 +269,16 @@ class AndroidExtraction(MVTModule):
         if header["encryption"] == "none":
             return parse_backup_file(backup_output, password=None)
 
-        # Backup encrypted. Request password from user.
         for password_retry in range(0, 3):
-            backup_password = getpass.getpass(prompt="Backup Password: ", stream=None)
+            backup_password = getpass.getpass(prompt="Backup password: ", stream=None)
             try:
                 decrypted_backup_tar = parse_backup_file(backup_output, backup_password)
                 return decrypted_backup_tar
             except InvalidBackupPassword:
-                self.log.info("Invalid backup password")
+                self.log.error("You provided the wrong password! Please try again...")
 
         self.log.warn("All attempts to decrypt backup with password failed!")
 
-    def run(self):
+    def run(self) -> None:
         """Run the main procedure."""
         raise NotImplementedError

mvt/android/modules/adb/chrome_history.py

@@ -1,5 +1,5 @@
 # Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021-2022 The MVT Project Authors.
+# Copyright (c) 2021-2022 Claudio Guarnieri.
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 
@@ -20,13 +20,14 @@ CHROME_HISTORY_PATH = "data/data/com.android.chrome/app_chrome/Default/History"
 class ChromeHistory(AndroidExtraction):
     """This module extracts records from Android's Chrome browsing history."""
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 serial=None, fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
-    def serialize(self, record):
+    def serialize(self, record: dict) -> None:
         return {
             "timestamp": record["isodate"],
             "module": self.__class__.__name__,
@@ -34,7 +35,7 @@ class ChromeHistory(AndroidExtraction):
             "data": f"{record['id']} - {record['url']} (visit ID: {record['visit_id']}, redirect source: {record['redirect_source']})"
         }
 
-    def check_indicators(self):
+    def check_indicators(self) -> None:
         if not self.indicators:
             return
 
@@ -42,7 +43,7 @@ class ChromeHistory(AndroidExtraction):
             if self.indicators.check_domain(result["url"]):
                 self.detected.append(result)
 
-    def _parse_db(self, db_path):
+    def _parse_db(self, db_path: str) -> None:
         """Parse a Chrome History database file.
 
         :param db_path: Path to the History database to process.
@@ -77,7 +78,7 @@ class ChromeHistory(AndroidExtraction):
 
         log.info("Extracted a total of %d history items", len(self.results))
 
-    def run(self):
+    def run(self) -> None:
         try:
             self._adb_process_file(os.path.join("/", CHROME_HISTORY_PATH),
                                    self._parse_db)

mvt/android/modules/adb/dumpsys_accessibility.py

@@ -1,5 +1,5 @@
 # Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021-2022 The MVT Project Authors.
+# Copyright (c) 2021-2022 Claudio Guarnieri.
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 
@@ -15,13 +15,14 @@ log = logging.getLogger(__name__)
 class DumpsysAccessibility(AndroidExtraction):
     """This module extracts stats on accessibility."""
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 serial=None, fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
-    def check_indicators(self):
+    def check_indicators(self) -> None:
         if not self.indicators:
             return
 
@@ -32,7 +33,7 @@ class DumpsysAccessibility(AndroidExtraction):
                 self.detected.append(result)
                 continue
 
-    def run(self):
+    def run(self) -> None:
         self._adb_connect()
         output = self._adb_command("dumpsys accessibility")
         self._adb_disconnect()

mvt/android/modules/adb/dumpsys_activities.py

@@ -1,5 +1,5 @@
 # Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021-2022 The MVT Project Authors.
+# Copyright (c) 2021-2022 Claudio Guarnieri.
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 
@@ -15,15 +15,16 @@ log = logging.getLogger(__name__)
 class DumpsysActivities(AndroidExtraction):
     """This module extracts details on receivers for risky activities."""
 
-    def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 serial=None, fast_mode=False, log=None, results=[]):
-        super().__init__(file_path=file_path, base_folder=base_folder,
-                         output_folder=output_folder, fast_mode=fast_mode,
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
                          log=log, results=results)
 
         self.results = results if results else {}
 
-    def check_indicators(self):
+    def check_indicators(self) -> None:
         if not self.indicators:
             return
 
@@ -35,7 +36,7 @@ class DumpsysActivities(AndroidExtraction):
                     self.detected.append({intent: activity})
                     continue
 
-    def run(self):
+    def run(self) -> None:
         self._adb_connect()
         output = self._adb_command("dumpsys package")
         self._adb_disconnect()

mvt/android/modules/adb/dumpsys_appops.py

@@ -0,0 +1,66 @@
+# Mobile Verification Toolkit (MVT)
+# Copyright (c) 2021-2022 Claudio Guarnieri.
+# Use of this software is governed by the MVT License 1.1 that can be found at
+#   https://license.mvt.re/1.1/
+
+import logging
+
+from mvt.android.parsers.dumpsys import parse_dumpsys_appops
+
+from .base import AndroidExtraction
+
+log = logging.getLogger(__name__)
+
+
+class DumpsysAppOps(AndroidExtraction):
+    """This module extracts records from App-op Manager."""
+
+    slug = "dumpsys_appops"
+
+    def __init__(self, file_path: str = None, target_path: str = None,
+                 results_path: str = None, fast_mode: bool = False,
+                 log: logging.Logger = None, results: list = []) -> None:
+        super().__init__(file_path=file_path, target_path=target_path,
+                         results_path=results_path, fast_mode=fast_mode,
+                         log=log, results=results)
+
+    def serialize(self, record: dict) -> None:
+        records = []
+        for perm in record["permissions"]:
+            if "entries" not in perm:
+                continue
+
+            for entry in perm["entries"]:
+                if "timestamp" in entry:
+                    records.append({
+                        "timestamp": entry["timestamp"],
+                        "module": self.__class__.__name__,
+                        "event": entry["access"],
+                        "data": f"{record['package_name']} access to {perm['name']}: {entry['access']}",
+                    })
+
+        return records
+
+    def check_indicators(self) -> None:
+        for result in self.results:
+            if self.indicators:
+                ioc = self.indicators.check_app_id(result.get("package_name"))
+                if ioc:
+                    result["matched_indicator"] = ioc
+                    self.detected.append(result)
+                    continue
+
+            for perm in result["permissions"]:
+                if perm["name"] == "REQUEST_INSTALL_PACKAGES" and perm["access"] == "allow":
+                    self.log.info("Package %s with REQUEST_INSTALL_PACKAGES permission",
+                                  result["package_name"])
+
+    def run(self) -> None:
+        self._adb_connect()
+        output = self._adb_command("dumpsys appops")
+        self._adb_disconnect()
+
+        self.results = parse_dumpsys_appops(output)
+
+        self.log.info("Extracted a total of %d records from app-ops manager",
+                      len(self.results))