Bumped version

Fix stalkerware STIX URL

.github/workflows/flake8.yml

@@ -0,0 +1,26 @@
+name: Flake8
+
+on:
+  push:
+    paths:
+      - '*.py'
+
+jobs:
+  flake8_py3:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Setup Python
+        uses: actions/setup-python@v1
+        with:
+          python-version: 3.9
+          architecture: x64
+      - name: Checkout
+        uses: actions/checkout@master
+      - name: Install flake8
+        run: pip install flake8
+      - name: Run flake8
+        uses: suo/flake8-github-action@releases/v1
+        with:
+          checkName: 'flake8_py3'   # NOTE: this needs to be the same as the job name
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/python-package.yml

@@ -1,7 +1,7 @@
 # This workflow will install Python dependencies, run tests and lint with a variety of Python versions
 # For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions
 
-name: Python package
+name: CI
 
 on:
   push:

.gitignore

@@ -133,4 +133,7 @@ dmypy.json
 *~
 
 # IDEA Dev Environment
-.idea
+.idea
+
+# Sublime Text project files
+*.sublime*

AUTHORS

@@ -1,7 +0,0 @@
-MVT was originally authored by Claudio Guarnieri <nex@nex.sx>.
-
-For an up-to-date list of all contributors visit:
-    https://github.com/mvt-project/mvt/graphs/contributors
-
-Or run:
-    git shortlog -s -n

README.md

@@ -6,6 +6,8 @@
 
 [![](https://img.shields.io/pypi/v/mvt)](https://pypi.org/project/mvt/)
 [![Documentation Status](https://readthedocs.org/projects/mvt/badge/?version=latest)](https://docs.mvt.re/en/latest/?badge=latest)
+[![CI](https://github.com/mvt-project/mvt/actions/workflows/python-package.yml/badge.svg)](https://github.com/mvt-project/mvt/actions/workflows/python-package.yml)
+[![Downloads](https://pepy.tech/badge/mvt)](https://pepy.tech/project/mvt)
 
 Mobile Verification Toolkit (MVT) is a collection of utilities to simplify and automate the process of gathering forensic traces helpful to identify a potential compromise of Android and iOS devices.
 
@@ -13,6 +15,7 @@ It has been developed and released by the [Amnesty International Security Lab](h
 
 *Warning*: MVT is a forensic research tool intended for technologists and investigators. Using it requires understanding the basics of forensic analysis and using command-line tools. This is not intended for end-user self-assessment. If you are concerned with the security of your device please seek expert assistance.
 
+
 ## Installation
 
 MVT can be installed from sources or from [PyPi](https://pypi.org/project/mvt/) (you will need some dependencies, check the [documentation](https://docs.mvt.re/en/latest/install/)):
@@ -21,14 +24,14 @@ MVT can be installed from sources or from [PyPi](https://pypi.org/project/mvt/)
 pip3 install mvt
 ```
 
-Alternatively, you can decide to run MVT and all relevant tools through a [Docker container](https://docs.mvt.re/en/latest/docker/).
+For alternative installation options and known issues, please refer to the [documentation](https://docs.mvt.re/en/latest/install/) as well as [GitHub Issues](https://github.com/mvt-project/mvt/issues).
 
-**Please note:** MVT is best run on Linux or Mac systems. [It does not currently support running natively on Windows.](https://docs.mvt.re/en/latest/install/#mvt-on-windows)
 
 ## Usage
 
 MVT provides two commands `mvt-ios` and `mvt-android`. [Check out the documentation to learn how to use them!](https://docs.mvt.re/)
 
+
 ## License
 
 The purpose of MVT is to facilitate the ***consensual forensic analysis*** of devices of those who might be targets of sophisticated mobile spyware attacks, especially members of civil society and marginalized communities. We do not want MVT to enable privacy violations of non-consenting individuals.  In order to achieve this, MVT is released under its own license. [Read more here.](https://docs.mvt.re/en/latest/license/)

dev/mvt-android

@@ -1,6 +1,6 @@
 #!/usr/bin/env python3
 # Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021-2022 The MVT Project Authors.
+# Copyright (c) 2021-2022 Claudio Guarnieri.
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 

dev/mvt-ios

@@ -1,6 +1,6 @@
 #!/usr/bin/env python3
 # Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021-2022 The MVT Project Authors.
+# Copyright (c) 2021-2022 Claudio Guarnieri.
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 

docs/android/backup.md

@@ -11,18 +11,37 @@ That said, most versions of Android should still allow to locally backup SMS mes
 Because `mvt-android check-backup` currently only supports checking SMS messages, you can indicate to backup only those:
 
 ```bash
-adb backup com.android.providers.telephony
+adb backup -nocompress com.android.providers.telephony
 ```
 
 In case you nonetheless wish to take a full backup, you can do so with
 
 ```bash
-adb backup -all
+adb backup -nocompress -all
 ```
 
-## Unpack the backup
+Some recent phones will enforce the utilisation of a password to encrypt the backup archive. In that case, the password will obviously be needed to extract and analyse the data later on.
 
-In order to unpack the backup, use [Android Backup Extractor (ABE)](https://github.com/nelenkov/android-backup-extractor) to convert it to a readable file format. Make sure that java is installed on your system and use the following command:
+## Unpack and check the backup
+
+MVT includes a partial implementation of the Android Backup parsing, because of the implementation difference in the compression algorithm between Java and Python. The `-nocompress` option passed to adb in the section above allows to avoid this issue. You can analyse and extract SMSs containing links from the backup directly with MVT:
+
+```bash
+$ mvt-android check-backup --output /path/to/results/ /path/to/backup.ab
+14:09:45 INFO     [mvt.android.cli] Checking ADB backup located at: backup.ab
+         INFO     [mvt.android.modules.backup.sms] Running module SMS...
+         INFO     [mvt.android.modules.backup.sms] Processing SMS backup file at
+                  apps/com.android.providers.telephony/d_f/000000_sms_backup
+         INFO     [mvt.android.modules.backup.sms] Extracted a total of 64 SMS messages containing links
+```
+
+If the backup is encrypted, MVT will prompt you to enter the password.
+
+Through the `--iocs` argument you can specify a [STIX2](https://oasis-open.github.io/cti-documentation/stix/intro) file defining a list of malicious indicators to check against the records extracted from the backup by MVT. Any matches will be highlighted in the terminal output.
+
+## Alternative ways to unpack and check the backup
+
+If you encounter an issue during the analysis of the backup, you can alternatively use [Android Backup Extractor (ABE)](https://github.com/nelenkov/android-backup-extractor) to convert it to a readable file format. Make sure that java is installed on your system and use the following command:
 
 ```bash
 java -jar ~/path/to/abe.jar unpack backup.ab backup.tar
@@ -33,17 +52,4 @@ If the backup is encrypted, ABE will prompt you to enter the password.
 
 Alternatively, [ab-decrypt](https://github.com/joernheissler/ab-decrypt) can be used for that purpose.
 
-## Check the backup
-
-You can then extract SMSs containing links with MVT:
-
-```bash
-$ mvt-android check-backup --output /path/to/results/ /path/to/backup/
-16:18:38 INFO     [mvt.android.cli] Checking ADB backup located at: .
-         INFO     [mvt.android.modules.backup.sms] Running module SMS...
-         INFO     [mvt.android.modules.backup.sms] Processing SMS backup file at /path/to/backup/apps/com.android.providers.telephony/d_f/000000_sms_backup
-16:18:39 INFO     [mvt.android.modules.backup.sms] Extracted a total of
-                  64 SMS messages containing links
-```
-
-Through the `--iocs` argument you can specify a [STIX2](https://oasis-open.github.io/cti-documentation/stix/intro) file defining a list of malicious indicators to check against the records extracted from the backup by MVT. Any matches will be highlighted in the terminal output.
+You can then extract SMSs containing links with MVT by passing the folder path as parameter instead of the `.ab` file: `mvt-android check-backup --output /path/to/results/ /path/to/backup/` (the path to backup given should be the folder containing the `apps` folder).

docs/iocs.md

@@ -39,7 +39,7 @@ export MVT_STIX2="/home/user/IOC1.stix2:/home/user/IOC2.stix2"
 - The [Amnesty International investigations repository](https://github.com/AmnestyTech/investigations) contains STIX-formatted IOCs for:
     - [Pegasus](https://en.wikipedia.org/wiki/Pegasus_(spyware)) ([STIX2](https://raw.githubusercontent.com/AmnestyTech/investigations/master/2021-07-18_nso/pegasus.stix2))
     - [Predator from Cytrox](https://citizenlab.ca/2021/12/pegasus-vs-predator-dissidents-doubly-infected-iphone-reveals-cytrox-mercenary-spyware/) ([STIX2](https://raw.githubusercontent.com/AmnestyTech/investigations/master/2021-12-16_cytrox/cytrox.stix2))
-- [This repository](https://github.com/Te-k/stalkerware-indicators) contains IOCs for Android stalkerware including [a STIX MVT-compatible file](https://raw.githubusercontent.com/Te-k/stalkerware-indicators/master/stalkerware.stix2).
+- [This repository](https://github.com/Te-k/stalkerware-indicators) contains IOCs for Android stalkerware including [a STIX MVT-compatible file](https://raw.githubusercontent.com/Te-k/stalkerware-indicators/master/generated/stalkerware.stix2).
 
 You can automaticallly download the latest public indicator files with the command `mvt-ios download-iocs` or `mvt-android download-iocs`. These commands download the list of indicators listed [here](https://github.com/mvt-project/mvt/blob/main/public_indicators.json) and store them in the [appdir](https://pypi.org/project/appdirs/) folder. They are then loaded automatically by mvt.
 

mvt/__init__.py

@@ -1,4 +1,4 @@
 # Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021-2022 The MVT Project Authors.
+# Copyright (c) 2021-2022 Claudio Guarnieri.
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/

mvt/android/__init__.py

@@ -1,5 +1,5 @@
 # Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021-2022 The MVT Project Authors.
+# Copyright (c) 2021-2022 Claudio Guarnieri.
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 

mvt/android/cli.py

@@ -1,14 +1,22 @@
 # Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021-2022 The MVT Project Authors.
+# Copyright (c) 2021-2022 Claudio Guarnieri.
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 
+import getpass
+import io
 import logging
 import os
+import tarfile
+from pathlib import Path
+from zipfile import ZipFile
 
 import click
 from rich.logging import RichHandler
 
+from mvt.android.parsers.backup import (AndroidBackupParsingError,
+                                        InvalidBackupPassword, parse_ab_header,
+                                        parse_backup_file)
 from mvt.common.help import (HELP_MSG_FAST, HELP_MSG_IOC,
                              HELP_MSG_LIST_MODULES, HELP_MSG_MODULE,
                              HELP_MSG_OUTPUT, HELP_MSG_SERIAL)
@@ -21,6 +29,7 @@ from .lookups.koodous import koodous_lookup
 from .lookups.virustotal import virustotal_lookup
 from .modules.adb import ADB_MODULES
 from .modules.backup import BACKUP_MODULES
+from .modules.bugreport import BUGREPORT_MODULES
 
 # Setup logging using Rich.
 LOG_FORMAT = "[%(name)s] %(message)s"
@@ -46,9 +55,9 @@ def version():
 
 
 #==============================================================================
-# Download APKs
+# Command: download-apks
 #==============================================================================
-@cli.command("download-apks", help="Download all or non-safelisted installed APKs installed on the device")
+@cli.command("download-apks", help="Download all or only non-system installed APKs")
 @click.option("--serial", "-s", type=str, help=HELP_MSG_SERIAL)
 @click.option("--all-apks", "-a", is_flag=True,
               help="Extract all packages installed on the phone, including system packages")
@@ -99,7 +108,7 @@ def download_apks(ctx, all_apks, virustotal, koodous, all_checks, output, from_f
 
 
 #==============================================================================
-# Checks through ADB
+# Command: check-adb
 #==============================================================================
 @cli.command("check-adb", help="Check an Android device over adb")
 @click.option("--serial", "-s", type=str, help=HELP_MSG_SERIAL)
@@ -157,17 +166,25 @@ def check_adb(ctx, iocs, output, fast, list_modules, module, serial):
 
 
 #==============================================================================
-# Check ADB backup
+# Command: check-bugreport
 #==============================================================================
-@cli.command("check-backup", help="Check an Android Backup")
-@click.option("--serial", "-s", type=str, help=HELP_MSG_SERIAL)
+@cli.command("check-bugreport", help="Check an Android Bug Report")
 @click.option("--iocs", "-i", type=click.Path(exists=True), multiple=True,
               default=[], help=HELP_MSG_IOC)
 @click.option("--output", "-o", type=click.Path(exists=False), help=HELP_MSG_OUTPUT)
-@click.argument("BACKUP_PATH", type=click.Path(exists=True))
+@click.option("--list-modules", "-l", is_flag=True, help=HELP_MSG_LIST_MODULES)
+@click.option("--module", "-m", help=HELP_MSG_MODULE)
+@click.argument("BUGREPORT_PATH", type=click.Path(exists=True))
 @click.pass_context
-def check_backup(ctx, iocs, output, backup_path, serial):
-    log.info("Checking ADB backup located at: %s", backup_path)
+def check_bugreport(ctx, iocs, output, list_modules, module, bugreport_path):
+    if list_modules:
+        log.info("Following is the list of available check-bugreport modules:")
+        for adb_module in BUGREPORT_MODULES:
+            log.info(" - %s", adb_module.__name__)
+
+        return
+
+    log.info("Checking an Android Bug Report located at: %s", bugreport_path)
 
     if output and not os.path.exists(output):
         try:
@@ -179,14 +196,110 @@ def check_backup(ctx, iocs, output, backup_path, serial):
     indicators = Indicators(log=log)
     indicators.load_indicators_files(iocs)
 
-    if os.path.isfile(backup_path):
-        log.critical("The path you specified is a not a folder!")
+    if os.path.isfile(bugreport_path):
+        bugreport_format = "zip"
+        zip_archive = ZipFile(bugreport_path)
+        zip_files = []
+        for file_name in zip_archive.namelist():
+            zip_files.append(file_name)
+    elif os.path.isdir(bugreport_path):
+        bugreport_format = "dir"
+        folder_files = []
+        parent_path = Path(bugreport_path).absolute().as_posix()
+        for root, subdirs, subfiles in os.walk(os.path.abspath(bugreport_path)):
+            for file_name in subfiles:
+                folder_files.append(os.path.relpath(os.path.join(root, file_name), parent_path))
 
-        if os.path.basename(backup_path) == "backup.ab":
-            log.info("You can use ABE (https://github.com/nelenkov/android-backup-extractor) "
-                     "to extract 'backup.ab' files!")
+    timeline = []
+    timeline_detected = []
+    for bugreport_module in BUGREPORT_MODULES:
+        if module and bugreport_module.__name__ != module:
+            continue
+
+        m = bugreport_module(base_folder=bugreport_path, output_folder=output,
+                             log=logging.getLogger(bugreport_module.__module__))
+
+        if bugreport_format == "zip":
+            m.from_zip(zip_archive, zip_files)
+        else:
+            m.from_folder(bugreport_path, folder_files)
+
+        if indicators.total_ioc_count:
+            m.indicators = indicators
+            m.indicators.log = m.log
+
+        run_module(m)
+        timeline.extend(m.timeline)
+        timeline_detected.extend(m.timeline_detected)
+
+    if output:
+        if len(timeline) > 0:
+            save_timeline(timeline, os.path.join(output, "timeline.csv"))
+        if len(timeline_detected) > 0:
+            save_timeline(timeline_detected, os.path.join(output, "timeline_detected.csv"))
+
+
+#==============================================================================
+# Command: check-backup
+#==============================================================================
+@cli.command("check-backup", help="Check an Android Backup")
+@click.option("--serial", "-s", type=str, help=HELP_MSG_SERIAL)
+@click.option("--iocs", "-i", type=click.Path(exists=True), multiple=True,
+              default=[], help=HELP_MSG_IOC)
+@click.option("--output", "-o", type=click.Path(exists=False), help=HELP_MSG_OUTPUT)
+@click.argument("BACKUP_PATH", type=click.Path(exists=True))
+@click.pass_context
+def check_backup(ctx, iocs, output, backup_path, serial):
+    log.info("Checking ADB backup located at: %s", backup_path)
+
+    if os.path.isfile(backup_path):
+        # AB File
+        backup_type = "ab"
+        with open(backup_path, "rb") as handle:
+            data = handle.read()
+        header = parse_ab_header(data)
+        if not header["backup"]:
+            log.critical("Invalid backup format, file should be in .ab format")
+            ctx.exit(1)
+        password = None
+        if header["encryption"] != "none":
+            password = getpass.getpass(prompt="Backup Password: ", stream=None)
+        try:
+            tardata = parse_backup_file(data, password=password)
+        except InvalidBackupPassword:
+            log.critical("Invalid backup password")
+            ctx.exit(1)
+        except AndroidBackupParsingError:
+            log.critical("Impossible to parse this backup file, please use Android Backup Extractor instead")
+            ctx.exit(1)
+
+        dbytes = io.BytesIO(tardata)
+        tar = tarfile.open(fileobj=dbytes)
+        files = []
+        for member in tar:
+            files.append(member.name)
+
+    elif os.path.isdir(backup_path):
+        backup_type = "folder"
+        backup_path = Path(backup_path).absolute().as_posix()
+        files = []
+        for root, subdirs, subfiles in os.walk(os.path.abspath(backup_path)):
+            for fname in subfiles:
+                files.append(os.path.relpath(os.path.join(root, fname), backup_path))
+    else:
+        log.critical("Invalid backup path, path should be a folder or an Android Backup (.ab) file")
         ctx.exit(1)
 
+    if output and not os.path.exists(output):
+        try:
+            os.makedirs(output)
+        except Exception as e:
+            log.critical("Unable to create output folder %s: %s", output, e)
+            ctx.exit(1)
+
+    indicators = Indicators(log=log)
+    indicators.load_indicators_files(iocs)
+
     for module in BACKUP_MODULES:
         m = module(base_folder=backup_path, output_folder=output,
                    log=logging.getLogger(module.__module__))
@@ -196,6 +309,11 @@ def check_backup(ctx, iocs, output, backup_path, serial):
         if serial:
             m.serial = serial
 
+        if backup_type == "folder":
+            m.from_folder(backup_path, files)
+        else:
+            m.from_ab(backup_path, tar, files)
+
         run_module(m)
 
 

mvt/android/data/root_binaries.txt

@@ -1,10 +0,0 @@
-su
-busybox
-supersu
-Superuser.apk
-KingoUser.apk
-SuperSu.apk
-magisk
-magiskhide
-magiskinit
-magiskpolicy

mvt/android/data/root_packages.txt

@@ -1,25 +0,0 @@
-com.noshufou.android.su
-com.noshufou.android.su.elite
-eu.chainfire.supersu
-com.koushikdutta.superuser
-com.thirdparty.superuser
-com.yellowes.su
-com.koushikdutta.rommanager
-com.koushikdutta.rommanager.license
-com.dimonvideo.luckypatcher
-com.chelpus.lackypatch
-com.ramdroid.appquarantine
-com.ramdroid.appquarantinepro
-com.devadvance.rootcloak
-com.devadvance.rootcloakplus
-de.robv.android.xposed.installer
-com.saurik.substrate
-com.zachspong.temprootremovejb
-com.amphoras.hidemyroot
-com.amphoras.hidemyrootadfree
-com.formyhm.hiderootPremium
-com.formyhm.hideroot
-me.phh.superuser
-eu.chainfire.supersu.pro
-com.kingouser.com
-com.topjohnwu.magisk

mvt/android/download_apks.py

@@ -1,5 +1,5 @@
 # Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021-2022 The MVT Project Authors.
+# Copyright (c) 2021-2022 Claudio Guarnieri.
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 

mvt/android/lookups/__init__.py

@@ -1,4 +1,4 @@
 # Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021-2022 The MVT Project Authors.
+# Copyright (c) 2021-2022 Claudio Guarnieri.
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/

mvt/android/lookups/koodous.py

@@ -1,5 +1,5 @@
 # Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021-2022 The MVT Project Authors.
+# Copyright (c) 2021-2022 Claudio Guarnieri.
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 

mvt/android/lookups/virustotal.py

@@ -1,5 +1,5 @@
 # Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021-2022 The MVT Project Authors.
+# Copyright (c) 2021-2022 Claudio Guarnieri.
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 

mvt/android/modules/__init__.py

@@ -1,4 +1,4 @@
 # Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021-2022 The MVT Project Authors.
+# Copyright (c) 2021-2022 Claudio Guarnieri.
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/

mvt/android/modules/adb/__init__.py

@@ -1,11 +1,12 @@
 # Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021-2022 The MVT Project Authors.
+# Copyright (c) 2021-2022 Claudio Guarnieri.
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 
 from .chrome_history import ChromeHistory
 from .dumpsys_accessibility import DumpsysAccessibility
 from .dumpsys_activities import DumpsysActivities
+from .dumpsys_appops import DumpsysAppOps
 from .dumpsys_battery_daily import DumpsysBatteryDaily
 from .dumpsys_battery_history import DumpsysBatteryHistory
 from .dumpsys_dbinfo import DumpsysDBInfo
@@ -17,11 +18,13 @@ from .logcat import Logcat
 from .packages import Packages
 from .processes import Processes
 from .root_binaries import RootBinaries
+from .selinux_status import SELinuxStatus
 from .settings import Settings
 from .sms import SMS
 from .whatsapp import Whatsapp
 
 ADB_MODULES = [ChromeHistory, SMS, Whatsapp, Processes, Getprop, Settings,
-               DumpsysBatteryHistory, DumpsysBatteryDaily, DumpsysReceivers,
-               DumpsysActivities, DumpsysAccessibility, DumpsysDBInfo,
-               DumpsysFull, Packages, RootBinaries, Logcat, Files]
+               SELinuxStatus, DumpsysBatteryHistory, DumpsysBatteryDaily,
+               DumpsysReceivers, DumpsysActivities, DumpsysAccessibility,
+               DumpsysDBInfo, DumpsysFull, DumpsysAppOps, Packages, Logcat,
+               RootBinaries, Files]

mvt/android/modules/adb/base.py

@@ -1,8 +1,10 @@
 # Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021-2022 The MVT Project Authors.
+# Copyright (c) 2021-2022 Claudio Guarnieri.
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 
+import base64
+import getpass
 import logging
 import os
 import random
@@ -18,6 +20,8 @@ from adb_shell.exceptions import (AdbCommandFailureException, DeviceAuthError,
                                   UsbDeviceNotFoundError, UsbReadFailedError)
 from usb1 import USBErrorAccess, USBErrorBusy
 
+from mvt.android.parsers.backup import (InvalidBackupPassword, parse_ab_header,
+                                        parse_backup_file)
 from mvt.common.module import InsufficientPrivileges, MVTModule
 
 log = logging.getLogger(__name__)
@@ -243,6 +247,32 @@ class AndroidExtraction(MVTModule):
         # Disconnect from the device.
         self._adb_disconnect()
 
+    def _generate_backup(self, package_name):
+        self.log.warning("Please check phone and accept Android backup prompt. You may need to set a backup password. \a")
+
+        # TODO: Base64 encoding as temporary fix to avoid byte-mangling over the shell transport...
+        backup_output_b64 = self._adb_command("/system/bin/bu backup -nocompress '{}' | base64".format(
+            package_name))
+        backup_output = base64.b64decode(backup_output_b64)
+        header = parse_ab_header(backup_output)
+
+        if not header["backup"]:
+            self.log.error("Extracting SMS via Android backup failed. No valid backup data found.")
+            return
+
+        if header["encryption"] == "none":
+            return parse_backup_file(backup_output, password=None)
+
+        for password_retry in range(0, 3):
+            backup_password = getpass.getpass(prompt="Backup password: ", stream=None)
+            try:
+                decrypted_backup_tar = parse_backup_file(backup_output, backup_password)
+                return decrypted_backup_tar
+            except InvalidBackupPassword:
+                self.log.error("You provided the wrong password! Please try again...")
+
+        self.log.warn("All attempts to decrypt backup with password failed!")
+
     def run(self):
         """Run the main procedure."""
         raise NotImplementedError

mvt/android/modules/adb/chrome_history.py

@@ -1,5 +1,5 @@
 # Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021-2022 The MVT Project Authors.
+# Copyright (c) 2021-2022 Claudio Guarnieri.
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 
@@ -68,7 +68,7 @@ class ChromeHistory(AndroidExtraction):
                 "url": item[1],
                 "visit_id": item[2],
                 "timestamp": item[3],
-                "isodate": convert_timestamp_to_iso(convert_chrometime_to_unix[item[3]]),
+                "isodate": convert_timestamp_to_iso(convert_chrometime_to_unix(item[3])),
                 "redirect_source": item[4],
             })
 
@@ -78,5 +78,8 @@ class ChromeHistory(AndroidExtraction):
         log.info("Extracted a total of %d history items", len(self.results))
 
     def run(self):
-        self._adb_process_file(os.path.join("/", CHROME_HISTORY_PATH),
-                               self._parse_db)
+        try:
+            self._adb_process_file(os.path.join("/", CHROME_HISTORY_PATH),
+                                   self._parse_db)
+        except Exception as e:
+            self.log.error(e)

mvt/android/modules/adb/dumpsys_accessibility.py

@@ -1,10 +1,12 @@
 # Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021-2022 The MVT Project Authors.
+# Copyright (c) 2021-2022 Claudio Guarnieri.
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 
 import logging
 
+from mvt.android.parsers import parse_dumpsys_accessibility
+
 from .base import AndroidExtraction
 
 log = logging.getLogger(__name__)
@@ -30,38 +32,14 @@ class DumpsysAccessibility(AndroidExtraction):
                 self.detected.append(result)
                 continue
 
-    @staticmethod
-    def parse_accessibility(output):
-        results = []
-
-        in_services = False
-        for line in output.split("\n"):
-            if line.strip().startswith("installed services:"):
-                in_services = True
-                continue
-
-            if not in_services:
-                continue
-
-            if line.strip() == "}":
-                break
-
-            service = line.split(":")[1].strip()
-            log.info("Found installed accessibility service \"%s\"", service)
-
-            results.append({
-                "package_name": service.split("/")[0],
-                "service": service,
-            })
-
-        return results
-
     def run(self):
         self._adb_connect()
-
         output = self._adb_command("dumpsys accessibility")
-        self.results = self.parse_accessibility(output)
+        self._adb_disconnect()
+
+        self.results = parse_dumpsys_accessibility(output)
+
+        for result in self.results:
+            log.info("Found installed accessibility service \"%s\"", result.get("service"))
 
         self.log.info("Identified a total of %d accessibility services", len(self.results))
-
-        self._adb_disconnect()

mvt/android/modules/adb/dumpsys_activities.py

@@ -1,10 +1,12 @@
 # Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021-2022 The MVT Project Authors.
+# Copyright (c) 2021-2022 Claudio Guarnieri.
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 
 import logging
 
+from mvt.android.parsers import parse_dumpsys_activity_resolver_table
+
 from .base import AndroidExtraction
 
 log = logging.getLogger(__name__)
@@ -33,66 +35,11 @@ class DumpsysActivities(AndroidExtraction):
                     self.detected.append({intent: activity})
                     continue
 
-    @staticmethod
-    def parse_activity_resolver_table(output):
-        results = {}
-
-        in_activity_resolver_table = False
-        in_non_data_actions = False
-        intent = None
-        for line in output.split("\n"):
-            if line.startswith("Activity Resolver Table:"):
-                in_activity_resolver_table = True
-                continue
-
-            if not in_activity_resolver_table:
-                continue
-
-            if line.startswith("  Non-Data Actions:"):
-                in_non_data_actions = True
-                continue
-
-            if not in_non_data_actions:
-                continue
-
-            # If we hit an empty line, the Non-Data Actions section should be
-            # finished.
-            if line.strip() == "":
-                break
-
-            # We detect the action name.
-            if line.startswith(" " * 6) and not line.startswith(" " * 8) and ":" in line:
-                intent = line.strip().replace(":", "")
-                results[intent] = []
-                continue
-
-            # If we are not in an intent block yet, skip.
-            if not intent:
-                continue
-
-            # If we are in a block but the line does not start with 8 spaces
-            # it means the block ended a new one started, so we reset and
-            # continue.
-            if not line.startswith(" " * 8):
-                intent = None
-                continue
-
-            # If we got this far, we are processing receivers for the
-            # activities we are interested in.
-            activity = line.strip().split(" ")[1]
-            package_name = activity.split("/")[0]
-
-            results[intent].append({
-                "package_name": package_name,
-                "activity": activity,
-            })
-
-        return results
-
     def run(self):
         self._adb_connect()
-
         output = self._adb_command("dumpsys package")
-        self.results = self.parse_activity_resolver_table(output)
-
         self._adb_disconnect()
+
+        self.results = parse_dumpsys_activity_resolver_table(output)
+
+        self.log.info("Extracted activities for %d intents", len(self.results))

mvt/android/modules/adb/dumpsys_appops.py

@@ -0,0 +1,66 @@
+# Mobile Verification Toolkit (MVT)
+# Copyright (c) 2021-2022 Claudio Guarnieri.
+# Use of this software is governed by the MVT License 1.1 that can be found at
+#   https://license.mvt.re/1.1/
+
+import logging
+import re
+
+from mvt.android.parsers.dumpsys import parse_dumpsys_appops
+
+from .base import AndroidExtraction
+
+log = logging.getLogger(__name__)
+
+
+class DumpsysAppOps(AndroidExtraction):
+    """This module extracts records from App-op Manager."""
+
+    slug = "dumpsys_appops"
+
+    def __init__(self, file_path=None, base_folder=None, output_folder=None,
+                 serial=None, fast_mode=False, log=None, results=[]):
+        super().__init__(file_path=file_path, base_folder=base_folder,
+                         output_folder=output_folder, fast_mode=fast_mode,
+                         log=log, results=results)
+
+    def serialize(self, record):
+        records = []
+        for perm in record["permissions"]:
+            if "entries" not in perm:
+                continue
+
+            for entry in perm["entries"]:
+                if "timestamp" in entry:
+                    records.append({
+                        "timestamp": entry["timestamp"],
+                        "module": self.__class__.__name__,
+                        "event": entry["access"],
+                        "data": f"{record['package_name']} access to {perm['name']} : {entry['access']}",
+                    })
+
+        return records
+
+    def check_indicators(self):
+        for result in self.results:
+            if self.indicators:
+                ioc = self.indicators.check_app_id(result.get("package_name"))
+                if ioc:
+                    result["matched_indicator"] = ioc
+                    self.detected.append(result)
+                    continue
+
+            for perm in result["permissions"]:
+                if perm["name"] == "REQUEST_INSTALL_PACKAGES" and perm["access"] == "allow":
+                    self.log.info("Package %s with REQUEST_INSTALL_PACKAGES permission",
+                                  result["package_name"])
+
+    def run(self):
+        self._adb_connect()
+        output = self._adb_command("dumpsys appops")
+        self._adb_disconnect()
+
+        self.results = parse_dumpsys_appops(output)
+
+        self.log.info("Extracted a total of %d records from app-ops manager",
+                      len(self.results))

mvt/android/modules/adb/dumpsys_battery_daily.py

@@ -1,10 +1,12 @@
 # Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021-2022 The MVT Project Authors.
+# Copyright (c) 2021-2022 Claudio Guarnieri.
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 
 import logging
 
+from mvt.android.parsers import parse_dumpsys_battery_daily
+
 from .base import AndroidExtraction
 
 log = logging.getLogger(__name__)
@@ -38,56 +40,11 @@ class DumpsysBatteryDaily(AndroidExtraction):
                 self.detected.append(result)
                 continue
 
-    @staticmethod
-    def parse_battery_history(output):
-        results = []
-        daily = None
-        daily_updates = []
-        for line in output.split("\n")[1:]:
-            if line.startswith("  Daily from "):
-                timeframe = line[13:].strip()
-                date_from, date_to = timeframe.strip(":").split(" to ", 1)
-                daily = {"from": date_from[0:10], "to": date_to[0:10]}
-
-            if not daily:
-                continue
-
-            if line.strip() == "":
-                results.extend(daily_updates)
-                daily = None
-                daily_updates = []
-                continue
-
-            if not line.strip().startswith("Update "):
-                continue
-
-            line = line.strip().replace("Update ", "")
-            package_name, vers = line.split(" ", 1)
-            vers_nr = vers.split("=", 1)[1]
-
-            already_seen = False
-            for update in daily_updates:
-                if package_name == update["package_name"] and vers_nr == update["vers"]:
-                    already_seen = True
-                    break
-
-            if not already_seen:
-                daily_updates.append({
-                    "action": "update",
-                    "from": daily["from"],
-                    "to": daily["to"],
-                    "package_name": package_name,
-                    "vers": vers_nr,
-                })
-
-        return results
-
     def run(self):
         self._adb_connect()
-
         output = self._adb_command("dumpsys batterystats --daily")
-        self.results = self.parse_battery_history(output)
+        self._adb_disconnect()
+
+        self.results = parse_dumpsys_battery_daily(output)
 
         self.log.info("Extracted %d records from battery daily stats", len(self.results))
-
-        self._adb_disconnect()

mvt/android/modules/adb/dumpsys_battery_history.py

@@ -1,10 +1,12 @@
 # Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021-2022 The MVT Project Authors.
+# Copyright (c) 2021-2022 Claudio Guarnieri.
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 
 import logging
 
+from mvt.android.parsers import parse_dumpsys_battery_history
+
 from .base import AndroidExtraction
 
 log = logging.getLogger(__name__)
@@ -30,62 +32,11 @@ class DumpsysBatteryHistory(AndroidExtraction):
                 self.detected.append(result)
                 continue
 
-    @staticmethod
-    def parse_battery_history(output):
-        results = []
-
-        for line in output.split("\n")[1:]:
-            if line.strip() == "":
-                break
-
-            time_elapsed, rest = line.strip().split(" ", 1)
-
-            start = line.find(" 100 ")
-            if start == -1:
-                continue
-
-            line = line[start+5:]
-
-            event = ""
-            if line.startswith("+job"):
-                event = "start_job"
-            elif line.startswith("-job"):
-                event = "end_job"
-            elif line.startswith("+running +wake_lock="):
-                event = "wake"
-            else:
-                continue
-
-            if event in ["start_job", "end_job"]:
-                uid = line[line.find("=")+1:line.find(":")]
-                service = line[line.find(":")+1:].strip('"')
-                package_name = service.split("/")[0]
-            elif event == "wake":
-                uid = line[line.find("=")+1:line.find(":")]
-                service = line[line.find("*walarm*:")+9:].split(" ")[0].strip('"').strip()
-                if service == "" or "/" not in service:
-                    continue
-
-                package_name = service.split("/")[0]
-            else:
-                continue
-
-            results.append({
-                "time_elapsed": time_elapsed,
-                "event": event,
-                "uid": uid,
-                "package_name": package_name,
-                "service": service,
-            })
-
-        return results
-
     def run(self):
         self._adb_connect()
-
         output = self._adb_command("dumpsys batterystats --history")
-        self.results = self.parse_battery_history(output)
+        self._adb_disconnect()
+
+        self.results = parse_dumpsys_battery_history(output)
 
         self.log.info("Extracted %d records from battery history", len(self.results))
-
-        self._adb_disconnect()

mvt/android/modules/adb/dumpsys_dbinfo.py

@@ -1,11 +1,13 @@
 # Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021-2022 The MVT Project Authors.
+# Copyright (c) 2021-2022 Claudio Guarnieri.
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 
 import logging
 import re
 
+from mvt.android.parsers import parse_dumpsys_dbinfo
+
 from .base import AndroidExtraction
 
 log = logging.getLogger(__name__)
@@ -35,47 +37,12 @@ class DumpsysDBInfo(AndroidExtraction):
                     self.detected.append(result)
                     continue
 
-    @staticmethod
-    def parse_dbinfo(output):
-        results = []
-
-        rxp = re.compile(r'.*\[([0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}\.[0-9]{3})\].*\[Pid:\((\d+)\)\](\w+).*sql\=\"(.+?)\".*path\=(.*?$)')
-
-        in_operations = False
-        for line in output.split("\n"):
-            if line.strip() == "Most recently executed operations:":
-                in_operations = True
-                continue
-
-            if not in_operations:
-                continue
-
-            if not line.startswith("        "):
-                in_operations = False
-                continue
-
-            matches = rxp.findall(line)
-            if not matches:
-                continue
-
-            match = matches[0]
-            results.append({
-                "isodate": match[0],
-                "pid": match[1],
-                "action": match[2],
-                "sql": match[3],
-                "path": match[4],
-            })
-
-        return results
-
     def run(self):
         self._adb_connect()
-
         output = self._adb_command("dumpsys dbinfo")
-        self.results = self.parse_dbinfo(output)
+        self._adb_disconnect()
+
+        self.results = parse_dumpsys_dbinfo(output)
 
         self.log.info("Extracted a total of %d records from database information",
                       len(self.results))
-
-        self._adb_disconnect()

mvt/android/modules/adb/dumpsys_full.py

@@ -1,5 +1,5 @@
 # Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021-2022 The MVT Project Authors.
+# Copyright (c) 2021-2022 Claudio Guarnieri.
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 

mvt/android/modules/adb/dumpsys_receivers.py

@@ -1,10 +1,12 @@
 # Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021-2022 The MVT Project Authors.
+# Copyright (c) 2021-2022 Claudio Guarnieri.
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/
 
 import logging
 
+from mvt.android.parsers import parse_dumpsys_receiver_resolver_table
+
 from .base import AndroidExtraction
 
 log = logging.getLogger(__name__)
@@ -55,66 +57,10 @@ class DumpsysReceivers(AndroidExtraction):
                 self.detected.append({intent: receiver})
                 continue
 
-    @staticmethod
-    def parse_receiver_resolver_table(output):
-        results = {}
-
-        in_receiver_resolver_table = False
-        in_non_data_actions = False
-        intent = None
-        for line in output.split("\n"):
-            if line.startswith("Receiver Resolver Table:"):
-                in_receiver_resolver_table = True
-                continue
-
-            if not in_receiver_resolver_table:
-                continue
-
-            if line.startswith("  Non-Data Actions:"):
-                in_non_data_actions = True
-                continue
-
-            if not in_non_data_actions:
-                continue
-
-            # If we hit an empty line, the Non-Data Actions section should be
-            # finished.
-            if line.strip() == "":
-                break
-
-            # We detect the action name.
-            if line.startswith(" " * 6) and not line.startswith(" " * 8) and ":" in line:
-                intent = line.strip().replace(":", "")
-                results[intent] = []
-                continue
-
-            # If we are not in an intent block yet, skip.
-            if not intent:
-                continue
-
-            # If we are in a block but the line does not start with 8 spaces
-            # it means the block ended a new one started, so we reset and
-            # continue.
-            if not line.startswith(" " * 8):
-                intent = None
-                continue
-
-            # If we got this far, we are processing receivers for the
-            # activities we are interested in.
-            receiver = line.strip().split(" ")[1]
-            package_name = receiver.split("/")[0]
-
-            results[intent].append({
-                "package_name": package_name,
-                "receiver": receiver,
-            })
-
-        return results
-
     def run(self):
         self._adb_connect()
 
         output = self._adb_command("dumpsys package")
-        self.results = self.parse_receiver_resolver_table(output)
+        self.results = parse_dumpsys_receiver_resolver_table(output)
 
         self._adb_disconnect()

mvt/android/modules/adb/files.py

@@ -1,5 +1,5 @@
 # Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021-2022 The MVT Project Authors.
+# Copyright (c) 2021-2022 Claudio Guarnieri.
 # Use of this software is governed by the MVT License 1.1 that can be found at
 #   https://license.mvt.re/1.1/