Bumped version

As discussed in #147

AUTHORS

@@ -0,0 +1,7 @@
+MVT was originally authored by Claudio Guarnieri <nex@nex.sx>.
+
+For an up-to-date list of all contributors visit:
+    https://github.com/mvt-project/mvt/graphs/contributors
+
+Or run:
+    git shortlog -s -n

README.md

@@ -15,38 +15,20 @@ It has been developed and released by the [Amnesty International Security Lab](h
 
 ## Installation
 
-MVT can be installed from sources or conveniently using:
+MVT can be installed from sources or from [PyPi](https://pypi.org/project/mvt/) (you will need some dependencies, check the [documentation](https://docs.mvt.re/en/latest/install.html)):
 
 ```
 pip3 install mvt
 ```
 
-You will need some dependencies, so please check the [documentation](https://docs.mvt.re/en/latest/install.html).
-
 Alternatively, you can decide to run MVT and all relevant tools through a [Docker container](https://docs.mvt.re/en/latest/docker.html).
 
+**Please note:** MVT is best run on Linux or Mac systems. [It does not currently support running natively on Windows.](https://docs.mvt.re/en/latest/install.html#mvt-on-windows)
 
 ## Usage
 
-MVT provides two commands `mvt-ios` and `mvt-android` with the following subcommands available:
-
-* `mvt-ios`:
-    * `check-backup`: Extract artifacts from an iTunes backup
-    * `check-fs`: Extract artifacts from a full filesystem dump
-    * `check-iocs`: Compare stored JSON results to provided indicators
-    * `decrypt-backup`:  Decrypt an encrypted iTunes backup
-    * `extract-key`: Extract decryption key from an iTunes backup
-* `mvt-android`:
-    * `check-backup`: Check an Android Backup
-    * `download-apks`: Download all or non-safelisted installed APKs
-
-Check out [the documentation to see how to use them](https://docs.mvt.re/).
-
+MVT provides two commands `mvt-ios` and `mvt-android`. [Check out the documentation to learn how to use them!](https://docs.mvt.re/).
 
 ## License
 
-The purpose of MVT is to facilitate the ***consensual forensic analysis*** of devices of those who might be targets of sophisticated mobile spyware attacks, especially members of civil society and marginalized communities. We do not want MVT to enable privacy violations of non-consenting individuals. Therefore, the goal of this license is to prohibit the use of MVT (and any other software licensed the same) for the purpose of *adversarial forensics*.
-
-In order to achieve this, MVT is released under an adaptation of [Mozilla Public License v2.0](https://www.mozilla.org/MPL). This modified license includes a new clause 3.0, "Consensual Use Restriction" which permits the use of the licensed software (and any *"Larger Work"* derived from it) exclusively with the explicit consent of the person/s whose data is being extracted and/or analysed (*"Data Owner"*).
-
-[Read the LICENSE](https://github.com/mvt-project/mvt/blob/main/LICENSE)
+The purpose of MVT is to facilitate the ***consensual forensic analysis*** of devices of those who might be targets of sophisticated mobile spyware attacks, especially members of civil society and marginalized communities. We do not want MVT to enable privacy violations of non-consenting individuals.  In order to achieve this, MVT is released under its own license. [Read more here.](https://docs.mvt.re/en/latest/license.html)

dev/mvt-android

@@ -1,8 +1,8 @@
 #!/usr/bin/env python3
 # Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021 MVT Project Developers.
-# See the file 'LICENSE' for usage and copying permissions, or find a copy at
-#   https://github.com/mvt-project/mvt/blob/main/LICENSE
+# Copyright (c) 2021 The MVT Project Authors.
+# Use of this software is governed by the MVT License 1.1 that can be found at
+#   https://license.mvt.re/1.1/
 
 import os
 import sys

dev/mvt-ios

@@ -1,8 +1,8 @@
 #!/usr/bin/env python3
 # Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021 MVT Project Developers.
-# See the file 'LICENSE' for usage and copying permissions, or find a copy at
-#   https://github.com/mvt-project/mvt/blob/main/LICENSE
+# Copyright (c) 2021 The MVT Project Authors.
+# Use of this software is governed by the MVT License 1.1 that can be found at
+#   https://license.mvt.re/1.1/
 
 import os
 import sys

docs/android/adb.md

@@ -0,0 +1,8 @@
+# Check over ADB
+
+TODO
+
+<!-- In order to use `mvt-android` you need to connect your Android device to your computer. You will then need to [enable USB debugging](https://developer.android.com/studio/debug/dev-options#enable>) on the Android device.
+
+If this is the first time you connect to this device, you will need to approve the authentication keys through a prompt that will appear on your Android device.
+ -->

docs/android/backup.md

@@ -1,36 +1,45 @@
-# Checking SMSs from Android backup
+# Check an Android Backup (SMS messages)
 
-Some attacks against Android phones are done by sending malicious links by SMS. The Android backup feature does not allow to gather much information that can be interesting for a forensic analysis, but it can be used to extract SMSs and check them with MVT.
+Android supports generating a backup archive of all the installed applications which supports it. However, over the years this functionality has been increasingly abandoned in favor of enabling users to remotely backup their personal data over the cloud. App developers can therefore decide to opt out from allowing the apps' data from being exported locally.
 
-To do so, you need to connect your Android device to your computer. You will then need to [enable USB debugging](https://developer.android.com/studio/debug/dev-options#enable>) on the Android device.
+At the time of writing, the Android Debug Bridge (adb) command to generate backups is still available but marked as deprecated.
 
-If this is the first time you connect to this device, you will need to approve the authentication keys through a prompt that will appear on your Android device.
+That said, most versions of Android should still allow to locally backup SMS messages, and since messages are still a prime vehicle for phishing and malware attacks, you might still want to take advantage of this functionality while it is supported.
 
-Then you can use adb to extract the backup for SMS only with the following command:
+## Generate a backup
+
+Because `mvt-android check-backup` currently only supports checking SMS messages, you can indicate to backup only those:
 
 ```bash
 adb backup com.android.providers.telephony
 ```
 
-You will need to approve the backup on the phone and potentially enter a password to encrypt the backup. The backup will then be stored in a file named `backup.ab`.
+In case you nonetheless wish to take a full backup, you can do so with
 
-You will need to use [Android Backup Extractor](https://github.com/nelenkov/android-backup-extractor) to convert it to a readable file format. Make sure that java is installed on your system and use the following command:
 ```bash
-java -jar ~/Download/abe.jar unpack backup.ab backup.tar
+adb backup -all
+```
+
+## Unpack the backup
+
+In order to reliable unpack th [Android Backup Extractor (ABE)](https://github.com/nelenkov/android-backup-extractor) to convert it to a readable file format. Make sure that java is installed on your system and use the following command:
+
+```bash
+java -jar ~/path/to/abe.jar unpack backup.ab backup.tar
 tar xvf backup.tar
 ```
 
-(If the backup is encrypted, the password will be asked by Android Backup Extractor).
+If the backup is encrypted, ABE will prompt you to enter the password.
+
+## Check the backup
 
 You can then extract SMSs containing links with MVT:
 
 ```bash
-$ mvt-android check-backup --output . .
+$ mvt-android check-backup --output /path/to/results/ /path/to/backup/
 16:18:38 INFO     [mvt.android.cli] Checking ADB backup located at: .
          INFO     [mvt.android.modules.backup.sms] Running module SMS...
-         INFO     [mvt.android.modules.backup.sms] Processing SMS backup
-                  file at ./apps/com.android.providers.telephony/d_f/000
-                  000_sms_backup
+         INFO     [mvt.android.modules.backup.sms] Processing SMS backup file at /path/to/backup/apps/com.android.providers.telephony/d_f/000000_sms_backup
 16:18:39 INFO     [mvt.android.modules.backup.sms] Extracted a total of
                   64 SMS messages containing links
 ```

docs/android/download_apks.md

@@ -1,15 +1,18 @@
 # Downloading APKs from an Android phone
 
-In order to use `mvt-android` you need to connect your Android device to your computer. You will then need to [enable USB debugging](https://developer.android.com/studio/debug/dev-options#enable>) on the Android device.
+MVT allows to attempt to download all available installed packages (APKs) in order to further inspect them and potentially identify any which might be malicious in nature.
 
-If this is the first time you connect to this device, you will need to approve the authentication keys through a prompt that will appear on your Android device.
-
-Now you can launch `mvt-android` and specify the `download-apks` command and the path to the folder where you want to store the extracted data:
+You can do so by launching the following command:
 
 ```bash
 mvt-android download-apks --output /path/to/folder
 ```
 
+It might take several minutes to complete.
+
+!!! info
+    MVT will likely warn you it was unable to download certain installed packages. There is no reason to be alarmed: this is typically expected behavior when MVT attempts to download a system package it has no privileges to access.
+
 Optionally, you can decide to enable lookups of the SHA256 hash of all the extracted APKs on [VirusTotal](https://www.virustotal.com) and/or [Koodous](https://koodous.com). While these lookups do not provide any conclusive assessment on all of the extracted APKs, they might highlight any known malicious ones:
 
 ```bash
@@ -22,3 +25,10 @@ Or, to launch all available lookups::
 ```bash
 mvt-android download-apks --output /path/to/folder --all-checks
 ```
+
+In case you have a previous extraction of APKs you want to later check against VirusTotal and Koodous, you can do so with the following arguments:
+
+```bash
+mvt-android download-apks --from-file /path/to/folder/apks.json --all-checks
+```
+

docs/android/methodology.md

@@ -1,8 +1,20 @@
 # Methodology for Android forensic
 
-For different technical reasons, it is more complex to do a forensic analysis of an Android phone.
+Unfortunately Android devices provide much less observability than their iOS cousins. Android stores very little diagnostic information useful to triage potential compromises, and because of this `mvt-android` capabilities are limited as well.
 
-Currently MVT allows to perform two different checks on an Android phone:
+However, not all is lost.
 
-* Download APKs installed in order to analyze them
-* Extract Android backup in order to look for suspicious SMS
+## Check installed Apps
+
+Because malware attacks over Android typically take the form of malicious or backdoored apps, the very first thing you might want to do is to extract and verify all installed Android packages and triage quickly if there are any which stand out as malicious or which might be atypical.
+
+While it is out of the scope of this documentation to dwell into details on how to analyze Android apps, MVT does allow to easily and automatically extract information about installed apps, download copies of them, and quickly lookup services such as [VirusTotal](https://www.virustotal.com) or [Koodous](https://www.koodous.com) which might quickly indicate known bad apps.
+
+
+## Check the device over Android Debug Bridge
+
+TODO
+
+## Check an Android Backup (SMS messages)
+
+TODO

docs/install.md

@@ -12,6 +12,8 @@ sudo apt install python3 python3-pip libusb-1.0-0 sqlite3
 
 *libusb-1.0-0* is not required if you intend to only use `mvt-ios` and not `mvt-android`.
 
+When working with Android devices you should additionally install [Android SDK Platform Tools](https://developer.android.com/studio/releases/platform-tools). If you prefer to install a package made available by your distribution of choice, please make sure the version is recent to ensure compatibility with modern Android devices.
+
 ## Dependencies on Mac
 
 Running MVT on Mac requires Xcode and [homebrew](https://brew.sh) to be installed.
@@ -24,6 +26,20 @@ brew install python3 libusb sqlite3
 
 *libusb* is not required if you intend to only use `mvt-ios` and not `mvt-android`.
 
+When working with Android devices you should additionally install Android SDK Platform Tools:
+
+```bash
+brew install --cask android-platform-tools
+```
+
+Or by downloading the [official binary releases](https://developer.android.com/studio/releases/platform-tools).
+
+## MVT on Windows
+
+MVT does not currently officially support running natively on Windows. While most functionality should work out of the box, there are known issues especially with `mvt-android`.
+
+It is recommended to try installing and running MVT from [Windows Subsystem Linux (WSL)](https://docs.microsoft.com/en-us/windows/wsl/about) and follow Linux installation instructions for your distribution of choice.
+
 ## Installing MVT
 
 If you haven't done so, you can add this to your `.bashrc` or `.zshrc` file in order to add locally installed Pypi binaries to your `$PATH`:
@@ -35,7 +51,7 @@ export PATH=$PATH:~/.local/bin
 Then you can install MVT directly from [pypi](https://pypi.org/project/mvt/)
 
 ```bash
-pip install mvt
+pip3 install mvt
 ```
 
 Or from the source code:

docs/iocs.md

@@ -22,7 +22,11 @@ After extracting forensics data from a device, you are also able to compare it w
 mvt-ios check-iocs --iocs ~/iocs/malware.stix2 /path/to/iphone/output/
 ```
 
-If you're looking for indicators of compromise for a specific piece of malware or adversary, please ask investigators or anti-malware researchers who have the relevant expertise for a STIX file.
+The `--iocs` option can be invoked multiple times to let MVT import multiple STIX2 files at once. For example:
+
+```bash
+mvt-ios check-backup --iocs ~/iocs/malware1.stix --iocs ~/iocs/malware2.stix2 /path/to/backup
+```
 
 ## Known repositories of STIX2 IOCs
 

docs/ios/records.md

@@ -4,6 +4,16 @@ In this page you can find a (reasonably) up-to-date breakdown of the files creat
 
 ## Records extracted by `check-fs` or `check-backup`
 
+### `backup_info.json`
+
+!!! info "Availabiliy"
+    Backup: :material-check:  
+    Full filesystem dump: :material-close:
+
+This JSON file is created by mvt-ios' `BackupInfo` module. The module extracts some details about the backup and the device, such as name, phone number, IMEI, product type and version.
+
+---
+
 ### `cache_files.json`
 
 !!! info "Availability"
@@ -50,6 +60,16 @@ If indicators a provided through the command-line, they are checked against the
 
 ---
 
+### `configuration_profiles.json`
+
+!!! info "Availability"
+    Backup: :material-check:  
+    Full filesystem dump: :material-close:
+
+This JSON file is created by mvt-ios' `ConfigurationProfiles` module. The module extracts details about iOS configuration profiles that have been installed on the device. These should include both default iOS as well as third-party profiles.
+
+---
+
 ### `contacts.json`
 
 !!! info "Availability"
@@ -150,6 +170,16 @@ If indicators are provided through the command-line, they are checked against th
 
 ---
 
+### `profile_events.json`
+
+!!! info "Availability"
+    Backup: :material-check:  
+    Full filesystem dump: :material-close:
+
+This JSON file is created by mvt-ios' `ProfileEvents` module. The module extracts a timeline of configuration profile operations. For example, it should indicate when a new profile was installed from the Settings app, or when one was removed.
+
+---
+
 ### `safari_browser_state.json`
 
 !!! info "Availability"
@@ -242,6 +272,18 @@ If indicators are provided through the command-line, they are checked against th
 
 ---
 
+### `webkit_resource_load_statistics.json`
+
+!!! info "Availability"
+    Backup: :material-check:  
+    Full filesystem dump: :material-check:
+
+This JSON file is created by mvt-ios `WebkitResourceLoadStatistics` module. The module extracts records from available WebKit ResourceLoadStatistics *observations.db* SQLite3 databases. These records should indicate domain names contacted by apps, including a timestamp.
+
+If indicators are provided through the command-line, they are checked against the extracted domain names. Any matches are stored in *webkit_resource_load_statistics_detected.json*.
+
+---
+
 ### `webkit_safari_view_service.json`
 
 !!! info "Availability"

mkdocs.yml

@@ -42,7 +42,8 @@ nav:
         - Records extracted by mvt-ios: "ios/records.md"
     - MVT for Android:
         - Android Forensic Methodology: "android/methodology.md"
-        - Check APKs: "android/download_apks.md"
-        - Check an Android Backup: "android/backup.md"
+        - Check over ADB: "android/adb.md"
+        - Check an Android Backup (SMS messages): "android/backup.md"
+        - Download APKs: "android/download_apks.md"
     - Indicators of Compromise: "iocs.md"
     - License: "license.md"

mvt/__init__.py

@@ -1,4 +1,4 @@
 # Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021 MVT Project Developers.
-# See the file 'LICENSE' for usage and copying permissions, or find a copy at
-#   https://github.com/mvt-project/mvt/blob/main/LICENSE
+# Copyright (c) 2021 The MVT Project Authors.
+# Use of this software is governed by the MVT License 1.1 that can be found at
+#   https://license.mvt.re/1.1/

mvt/android/__init__.py

@@ -1,6 +1,6 @@
 # Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021 MVT Project Developers.
-# See the file 'LICENSE' for usage and copying permissions, or find a copy at
-#   https://github.com/mvt-project/mvt/blob/main/LICENSE
+# Copyright (c) 2021 The MVT Project Authors.
+# Use of this software is governed by the MVT License 1.1 that can be found at
+#   https://license.mvt.re/1.1/
 
 from .cli import cli

mvt/android/cli.py

@@ -1,17 +1,15 @@
 # Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021 MVT Project Developers.
-# See the file 'LICENSE' for usage and copying permissions, or find a copy at
-#   https://github.com/mvt-project/mvt/blob/main/LICENSE
+# Copyright (c) 2021 The MVT Project Authors.
+# Use of this software is governed by the MVT License 1.1 that can be found at
+#   https://license.mvt.re/1.1/
 
-import argparse
 import logging
 import os
-import sys
 
 import click
 from rich.logging import RichHandler
 
-from mvt.common.indicators import Indicators
+from mvt.common.indicators import Indicators, IndicatorsFileBadFormat
 from mvt.common.module import run_module, save_timeline
 
 from .download_apks import DownloadAPKs
@@ -28,7 +26,7 @@ log = logging.getLogger(__name__)
 
 # Help messages of repeating options.
 OUTPUT_HELP_MESSAGE = "Specify a path to a folder where you want to store JSON results"
-
+SERIAL_HELP_MESSAGE = "Specify a device serial number or HOST:PORT connection string"
 
 #==============================================================================
 # Main
@@ -42,6 +40,7 @@ def cli():
 # Download APKs
 #==============================================================================
 @cli.command("download-apks", help="Download all or non-safelisted installed APKs installed on the device")
+@click.option("--serial", "-s", type=str, help=SERIAL_HELP_MESSAGE)
 @click.option("--all-apks", "-a", is_flag=True,
               help="Extract all packages installed on the phone, even those marked as safe")
 @click.option("--virustotal", "-v", is_flag=True, help="Check packages on VirusTotal")
@@ -51,19 +50,27 @@ def cli():
               help="Specify a path to a folder where you want to store the APKs")
 @click.option("--from-file", "-f", type=click.Path(exists=True),
               help="Instead of acquiring from phone, load an existing packages.json file for lookups (mainly for debug purposes)")
-def download_apks(all_apks, virustotal, koodous, all_checks, output, from_file):
+@click.pass_context
+def download_apks(ctx, all_apks, virustotal, koodous, all_checks, output, from_file, serial):
     try:
         if from_file:
             download = DownloadAPKs.from_json(from_file)
         else:
-            if output and not os.path.exists(output):
+            # TODO: Do we actually want to be able to run without storing any file?
+            if not output:
+                log.critical("You need to specify an output folder with --output!")
+                ctx.exit(1)
+
+            if not os.path.exists(output):
                 try:
                     os.makedirs(output)
                 except Exception as e:
                     log.critical("Unable to create output folder %s: %s", output, e)
-                    sys.exit(-1)
+                    ctx.exit(1)
 
             download = DownloadAPKs(output_folder=output, all_apks=all_apks)
+            if serial:
+                download.serial = serial
             download.run()
 
         packages = download.packages
@@ -78,19 +85,22 @@ def download_apks(all_apks, virustotal, koodous, all_checks, output, from_file):
             koodous_lookup(packages)
     except KeyboardInterrupt:
         print("")
-        sys.exit(-1)
+        ctx.exit(1)
 
 
 #==============================================================================
 # Checks through ADB
 #==============================================================================
 @cli.command("check-adb", help="Check an Android device over adb")
-@click.option("--iocs", "-i", type=click.Path(exists=True), help="Path to indicators file")
+@click.option("--serial", "-s", type=str, help=SERIAL_HELP_MESSAGE)
+@click.option("--iocs", "-i", type=click.Path(exists=True), multiple=True,
+              default=[], help="Path to indicators file (can be invoked multiple times)")
 @click.option("--output", "-o", type=click.Path(exists=False),
               help="Specify a path to a folder where you want to store JSON results")
 @click.option("--list-modules", "-l", is_flag=True, help="Print list of available modules and exit")
 @click.option("--module", "-m", help="Name of a single module you would like to run instead of all")
-def check_adb(iocs, output, list_modules, module):
+@click.pass_context
+def check_adb(ctx, iocs, output, list_modules, module, serial):
     if list_modules:
         log.info("Following is the list of available check-adb modules:")
         for adb_module in ADB_MODULES:
@@ -105,12 +115,16 @@ def check_adb(iocs, output, list_modules, module):
             os.makedirs(output)
         except Exception as e:
             log.critical("Unable to create output folder %s: %s", output, e)
-            sys.exit(-1)
+            ctx.exit(1)
 
-    if iocs:
-        # Pre-load indicators for performance reasons.
-        log.info("Loading indicators from provided file at %s", iocs)
-        indicators = Indicators(iocs)
+    indicators = Indicators(log=log)
+    for ioc_path in iocs:
+        try:
+            indicators.parse_stix2(ioc_path)
+        except IndicatorsFileBadFormat as e:
+            log.critical(e)
+            ctx.exit(1)
+    log.info("Loaded a total of %d indicators", indicators.ioc_count)
 
     timeline = []
     timeline_detected = []
@@ -119,6 +133,8 @@ def check_adb(iocs, output, list_modules, module):
             continue
 
         m = adb_module(output_folder=output, log=logging.getLogger(adb_module.__module__))
+        if serial:
+            m.serial = serial
 
         if iocs:
             indicators.log = m.log
@@ -134,14 +150,18 @@ def check_adb(iocs, output, list_modules, module):
         if len(timeline_detected) > 0:
             save_timeline(timeline_detected, os.path.join(output, "timeline_detected.csv"))
 
+
 #==============================================================================
 # Check ADB backup
 #==============================================================================
 @cli.command("check-backup", help="Check an Android Backup")
-@click.option("--iocs", "-i", type=click.Path(exists=True), help="Path to indicators file")
+@click.option("--serial", "-s", type=str, help=SERIAL_HELP_MESSAGE)
+@click.option("--iocs", "-i", type=click.Path(exists=True), multiple=True,
+              default=[], help="Path to indicators file (can be invoked multiple times)")
 @click.option("--output", "-o", type=click.Path(exists=False), help=OUTPUT_HELP_MESSAGE)
 @click.argument("BACKUP_PATH", type=click.Path(exists=True))
-def check_backup(iocs, output, backup_path):
+@click.pass_context
+def check_backup(ctx, iocs, output, backup_path, serial):
     log.info("Checking ADB backup located at: %s", backup_path)
 
     if output and not os.path.exists(output):
@@ -149,12 +169,16 @@ def check_backup(iocs, output, backup_path):
             os.makedirs(output)
         except Exception as e:
             log.critical("Unable to create output folder %s: %s", output, e)
-            sys.exit(-1)
+            ctx.exit(1)
 
-    if iocs:
-        # Pre-load indicators for performance reasons.
-        log.info("Loading indicators from provided file at %s", iocs)
-        indicators = Indicators(iocs)
+    indicators = Indicators(log=log)
+    for ioc_path in iocs:
+        try:
+            indicators.parse_stix2(ioc_path)
+        except IndicatorsFileBadFormat as e:
+            log.critical(e)
+            ctx.exit(1)
+    log.info("Loaded a total of %d indicators", indicators.ioc_count)
 
     if os.path.isfile(backup_path):
         log.critical("The path you specified is a not a folder!")
@@ -162,12 +186,15 @@ def check_backup(iocs, output, backup_path):
         if os.path.basename(backup_path) == "backup.ab":
             log.info("You can use ABE (https://github.com/nelenkov/android-backup-extractor) " \
                      "to extract 'backup.ab' files!")
-        sys.exit(-1)
+        ctx.exit(1)
 
     for module in BACKUP_MODULES:
         m = module(base_folder=backup_path, output_folder=output,
                    log=logging.getLogger(module.__module__))
 
+        if serial:
+            m.serial = serial
+
         if iocs:
             indicators.log = m.log
             m.indicators = indicators

mvt/android/download_apks.py

@@ -1,7 +1,7 @@
 # Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021 MVT Project Developers.
-# See the file 'LICENSE' for usage and copying permissions, or find a copy at
-#   https://github.com/mvt-project/mvt/blob/main/LICENSE
+# Copyright (c) 2021 The MVT Project Authors.
+# Use of this software is governed by the MVT License 1.1 that can be found at
+#   https://license.mvt.re/1.1/
 
 import json
 import logging
@@ -10,6 +10,7 @@ import os
 import pkg_resources
 from tqdm import tqdm
 
+from mvt.common.module import InsufficientPrivileges
 from mvt.common.utils import get_sha256_from_file_path
 
 from .modules.adb.base import AndroidExtraction
@@ -58,8 +59,8 @@ class DownloadAPKs(AndroidExtraction):
 
     @classmethod
     def from_json(cls, json_path):
-        """Initialize this class from an existing packages.json file.
-        :param json_path: Path to the packages.json file to parse.
+        """Initialize this class from an existing apks.json file.
+        :param json_path: Path to the apks.json file to parse.
         """
         with open(json_path, "r") as handle:
             data = json.load(handle)
@@ -139,6 +140,11 @@ class DownloadAPKs(AndroidExtraction):
                               miniters=1) as pp:
                 self._adb_download(remote_path, local_path,
                                    progress_callback=pp.update_to)
+        except InsufficientPrivileges:
+            log.warn("Unable to pull package file from %s: insufficient privileges, it might be a system app",
+                     remote_path)
+            self._adb_reconnect()
+            return None
         except Exception as e:
             log.exception("Failed to pull package file from %s: %s",
                           remote_path, e)
@@ -196,7 +202,7 @@ class DownloadAPKs(AndroidExtraction):
     def save_json(self):
         """Save the results to the package.json file.
         """
-        json_path = os.path.join(self.output_folder, "packages.json")
+        json_path = os.path.join(self.output_folder, "apks.json")
         packages = []
         for package in self.packages:
             packages.append(package.__dict__)

mvt/android/lookups/__init__.py

@@ -1,4 +1,4 @@
 # Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021 MVT Project Developers.
-# See the file 'LICENSE' for usage and copying permissions, or find a copy at
-#   https://github.com/mvt-project/mvt/blob/main/LICENSE
+# Copyright (c) 2021 The MVT Project Authors.
+# Use of this software is governed by the MVT License 1.1 that can be found at
+#   https://license.mvt.re/1.1/

mvt/android/lookups/koodous.py

@@ -1,7 +1,7 @@
 # Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021 MVT Project Developers.
-# See the file 'LICENSE' for usage and copying permissions, or find a copy at
-#   https://github.com/mvt-project/mvt/blob/main/LICENSE
+# Copyright (c) 2021 The MVT Project Authors.
+# Use of this software is governed by the MVT License 1.1 that can be found at
+#   https://license.mvt.re/1.1/
 
 import logging
 

mvt/android/lookups/virustotal.py

@@ -1,7 +1,7 @@
 # Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021 MVT Project Developers.
-# See the file 'LICENSE' for usage and copying permissions, or find a copy at
-#   https://github.com/mvt-project/mvt/blob/main/LICENSE
+# Copyright (c) 2021 The MVT Project Authors.
+# Use of this software is governed by the MVT License 1.1 that can be found at
+#   https://license.mvt.re/1.1/
 
 import logging
 

mvt/android/modules/__init__.py

@@ -1,4 +1,4 @@
 # Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021 MVT Project Developers.
-# See the file 'LICENSE' for usage and copying permissions, or find a copy at
-#   https://github.com/mvt-project/mvt/blob/main/LICENSE
+# Copyright (c) 2021 The MVT Project Authors.
+# Use of this software is governed by the MVT License 1.1 that can be found at
+#   https://license.mvt.re/1.1/

mvt/android/modules/adb/__init__.py

@@ -1,7 +1,7 @@
 # Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021 MVT Project Developers.
-# See the file 'LICENSE' for usage and copying permissions, or find a copy at
-#   https://github.com/mvt-project/mvt/blob/main/LICENSE
+# Copyright (c) 2021 The MVT Project Authors.
+# Use of this software is governed by the MVT License 1.1 that can be found at
+#   https://license.mvt.re/1.1/
 
 from .chrome_history import ChromeHistory
 from .dumpsys_batterystats import DumpsysBatterystats

mvt/android/modules/adb/base.py

@@ -1,7 +1,7 @@
 # Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021 MVT Project Developers.
-# See the file 'LICENSE' for usage and copying permissions, or find a copy at
-#   https://github.com/mvt-project/mvt/blob/main/LICENSE
+# Copyright (c) 2021 The MVT Project Authors.
+# Use of this software is governed by the MVT License 1.1 that can be found at
+#   https://license.mvt.re/1.1/
 
 import logging
 import os
@@ -11,13 +11,14 @@ import sys
 import tempfile
 import time
 
-from adb_shell.adb_device import AdbDeviceUsb
+from adb_shell.adb_device import AdbDeviceTcp, AdbDeviceUsb
 from adb_shell.auth.keygen import keygen, write_public_keyfile
 from adb_shell.auth.sign_pythonrsa import PythonRSASigner
-from adb_shell.exceptions import AdbCommandFailureException, DeviceAuthError
+from adb_shell.exceptions import (AdbCommandFailureException, DeviceAuthError,
+                                  UsbReadFailedError)
 from usb1 import USBErrorAccess, USBErrorBusy
 
-from mvt.common.module import MVTModule
+from mvt.common.module import InsufficientPrivileges, MVTModule
 
 log = logging.getLogger(__name__)
 
@@ -29,17 +30,12 @@ class AndroidExtraction(MVTModule):
 
     def __init__(self, file_path=None, base_folder=None, output_folder=None,
                  fast_mode=False, log=None, results=[]):
-        """Initialize Android extraction module.
-        :param file_path: Path to the database file to parse
-        :param base_folder: Path to a base folder containing an Android dump
-        :param output_folder: Path to the folder where to store extraction
-                              results
-        """
         super().__init__(file_path=file_path, base_folder=base_folder,
                          output_folder=output_folder, fast_mode=fast_mode,
                          log=log, results=results)
 
         self.device = None
+        self.serial = None
 
     def _adb_check_keys(self):
         """Make sure Android adb keys exist.
@@ -59,7 +55,19 @@ class AndroidExtraction(MVTModule):
             priv_key = handle.read()
 
         signer = PythonRSASigner("", priv_key)
-        self.device = AdbDeviceUsb()
+
+        # If no serial was specified or if the serial does not seem to be
+        # a HOST:PORT definition, we use the USB transport.
+        if not self.serial or ":" not in self.serial:
+            self.device = AdbDeviceUsb(serial=self.serial)
+        # Otherwise we try to use the TCP transport.
+        else:
+            addr = self.serial.split(":")
+            if len(addr) < 2:
+                raise ValueError("TCP serial number must follow the format: `address:port`")
+
+            self.device = AdbDeviceTcp(addr[0], int(addr[1]),
+                                       default_transport_timeout_s=30.)
 
         while True:
             try:
@@ -70,9 +78,14 @@ class AndroidExtraction(MVTModule):
             except DeviceAuthError:
                 log.error("You need to authorize this computer on the Android device. Retrying in 5 seconds...")
                 time.sleep(5)
-            except Exception as e:
-                log.critical(e)
+            except UsbReadFailedError:
+                log.error("Unable to connect to the device over USB. Try to unplug, plug the device and start again.")
                 sys.exit(-1)
+            except OSError as e:
+                if e.errno == 113 and self.serial:
+                    log.critical("Unable to connect to the device %s: did you specify the correct IP addres?",
+                              self.serial)
+                    sys.exit(-1)
             else:
                 break
 
@@ -105,7 +118,7 @@ class AndroidExtraction(MVTModule):
         """Check if we have a `su` binary, otherwise raise an Exception.
         """
         if not self._adb_check_if_root():
-            raise Exception("The Android device does not seem to have a `su` binary. Cannot run this module.")
+            raise InsufficientPrivileges("This module is optionally available in case the device is already rooted. Do NOT root your own device!")
 
     def _adb_command_as_root(self, command):
         """Execute an adb shell command.
@@ -120,6 +133,8 @@ class AndroidExtraction(MVTModule):
         :returns: Boolean indicating whether the file exists or not
         """
 
+        # TODO: Need to support checking files without root privileges as well.
+
         # Connect to the device over adb.
         self._adb_connect()
         # Check if we have root, if not raise an Exception.

mvt/android/modules/adb/chrome_history.py

@@ -1,7 +1,7 @@
 # Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021 MVT Project Developers.
-# See the file 'LICENSE' for usage and copying permissions, or find a copy at
-#   https://github.com/mvt-project/mvt/blob/main/LICENSE
+# Copyright (c) 2021 The MVT Project Authors.
+# Use of this software is governed by the MVT License 1.1 that can be found at
+#   https://license.mvt.re/1.1/
 
 import logging
 import os
@@ -20,7 +20,7 @@ class ChromeHistory(AndroidExtraction):
     """This module extracts records from Android's Chrome browsing history."""
 
     def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 fast_mode=False, log=None, results=[]):
+                 serial=None, fast_mode=False, log=None, results=[]):
         super().__init__(file_path=file_path, base_folder=base_folder,
                          output_folder=output_folder, fast_mode=fast_mode,
                          log=log, results=results)
@@ -52,14 +52,14 @@ class ChromeHistory(AndroidExtraction):
         """)
 
         for item in cur:
-            self.results.append(dict(
-                id=item[0],
-                url=item[1],
-                visit_id=item[2],
-                timestamp=item[3],
-                isodate=convert_timestamp_to_iso(convert_chrometime_to_unix(item[3])),
-                redirect_source=item[4],
-            ))
+            self.results.append({
+                "id": item[0],
+                "url": item[1],
+                "visit_id": item[2],
+                "timestamp": item[3],
+                "isodate": convert_timestamp_to_iso(convert_chrometime_to_unix[item[3]]),
+                "redirect_source": item[4],
+            })
 
         cur.close()
         conn.close()

mvt/android/modules/adb/dumpsys_batterystats.py

@@ -1,7 +1,7 @@
 # Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021 MVT Project Developers.
-# See the file 'LICENSE' for usage and copying permissions, or find a copy at
-#   https://github.com/mvt-project/mvt/blob/main/LICENSE
+# Copyright (c) 2021 The MVT Project Authors.
+# Use of this software is governed by the MVT License 1.1 that can be found at
+#   https://license.mvt.re/1.1/
 
 import logging
 import os
@@ -14,7 +14,7 @@ class DumpsysBatterystats(AndroidExtraction):
     """This module extracts stats on battery consumption by processes."""
 
     def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 fast_mode=False, log=None, results=[]):
+                 serial=None, fast_mode=False, log=None, results=[]):
         super().__init__(file_path=file_path, base_folder=base_folder,
                          output_folder=output_folder, fast_mode=fast_mode,
                          log=log, results=results)

mvt/android/modules/adb/dumpsys_packages.py

@@ -1,7 +1,7 @@
 # Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021 MVT Project Developers.
-# See the file 'LICENSE' for usage and copying permissions, or find a copy at
-#   https://github.com/mvt-project/mvt/blob/main/LICENSE
+# Copyright (c) 2021 The MVT Project Authors.
+# Use of this software is governed by the MVT License 1.1 that can be found at
+#   https://license.mvt.re/1.1/
 
 import logging
 import os
@@ -14,7 +14,7 @@ class DumpsysPackages(AndroidExtraction):
     """This module extracts stats on installed packages."""
 
     def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 fast_mode=False, log=None, results=[]):
+                 serial=None, fast_mode=False, log=None, results=[]):
         super().__init__(file_path=file_path, base_folder=base_folder,
                          output_folder=output_folder, fast_mode=fast_mode,
                          log=log, results=results)

mvt/android/modules/adb/dumpsys_procstats.py

@@ -1,7 +1,7 @@
 # Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021 MVT Project Developers.
-# See the file 'LICENSE' for usage and copying permissions, or find a copy at
-#   https://github.com/mvt-project/mvt/blob/main/LICENSE
+# Copyright (c) 2021 The MVT Project Authors.
+# Use of this software is governed by the MVT License 1.1 that can be found at
+#   https://license.mvt.re/1.1/
 
 import logging
 import os
@@ -14,7 +14,7 @@ class DumpsysProcstats(AndroidExtraction):
     """This module extracts stats on memory consumption by processes."""
 
     def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 fast_mode=False, log=None, results=[]):
+                 serial=None, fast_mode=False, log=None, results=[]):
         super().__init__(file_path=file_path, base_folder=base_folder,
                          output_folder=output_folder, fast_mode=fast_mode,
                          log=log, results=results)

mvt/android/modules/adb/packages.py

@@ -1,7 +1,7 @@
 # Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021 MVT Project Developers.
-# See the file 'LICENSE' for usage and copying permissions, or find a copy at
-#   https://github.com/mvt-project/mvt/blob/main/LICENSE
+# Copyright (c) 2021 The MVT Project Authors.
+# Use of this software is governed by the MVT License 1.1 that can be found at
+#   https://license.mvt.re/1.1/
 
 import logging
 import os
@@ -16,7 +16,7 @@ class Packages(AndroidExtraction):
     """This module extracts the list of installed packages."""
 
     def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 fast_mode=False, log=None, results=[]):
+                 serial=None, fast_mode=False, log=None, results=[]):
         super().__init__(file_path=file_path, base_folder=base_folder,
                          output_folder=output_folder, fast_mode=fast_mode,
                          log=log, results=results)
@@ -66,9 +66,15 @@ class Packages(AndroidExtraction):
 
             fields = line.split()
             file_name, package_name = fields[0].split(":")[1].rsplit("=", 1)
-            installer = fields[1].split("=")[1].strip()
-            if installer == "null":
+
+            try:
+                installer = fields[1].split("=")[1].strip()
+            except IndexError:
                 installer = None
+            else:
+                if installer == "null":
+                    installer = None
+
             uid = fields[2].split(":")[1].strip()
 
             dumpsys = self._adb_command(f"dumpsys package {package_name} | grep -A2 timeStamp").split("\n")
@@ -76,18 +82,18 @@ class Packages(AndroidExtraction):
             first_install = dumpsys[1].split("=")[1].strip()
             last_update = dumpsys[2].split("=")[1].strip()
 
-            self.results.append(dict(
-                package_name=package_name,
-                file_name=file_name,
-                installer=installer,
-                timestamp=timestamp,
-                first_install_time=first_install,
-                last_update_time=last_update,
-                uid=uid,
-                disabled=False,
-                system=False,
-                third_party=False,
-            ))
+            self.results.append({
+                "package_name": package_name,
+                "file_name": file_name,
+                "installer": installer,
+                "timestamp": timestamp,
+                "first_install_time": first_install,
+                "last_update_time": last_update,
+                "uid": uid,
+                "disabled": False,
+                "system": False,
+                "third_party": False,
+            })
 
         cmds = [
             {"field": "disabled", "arg": "-d"},
@@ -106,6 +112,13 @@ class Packages(AndroidExtraction):
                     if result["package_name"] == package_name:
                         self.results[i][cmd["field"]] = True
 
+        for result in self.results:
+            if result["system"]:
+                continue
+
+            self.log.info("Found non-system package with name \"%s\" installed by \"%s\" on %s",
+                          result["package_name"], result["installer"], result["timestamp"])
+
         self.log.info("Extracted at total of %d installed package names",
                       len(self.results))
 

mvt/android/modules/adb/processes.py

@@ -1,7 +1,7 @@
 # Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021 MVT Project Developers.
-# See the file 'LICENSE' for usage and copying permissions, or find a copy at
-#   https://github.com/mvt-project/mvt/blob/main/LICENSE
+# Copyright (c) 2021 The MVT Project Authors.
+# Use of this software is governed by the MVT License 1.1 that can be found at
+#   https://license.mvt.re/1.1/
 
 import logging
 
@@ -13,7 +13,7 @@ class Processes(AndroidExtraction):
     """This module extracts details on running processes."""
 
     def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 fast_mode=False, log=None, results=[]):
+                 serial=None, fast_mode=False, log=None, results=[]):
         super().__init__(file_path=file_path, base_folder=base_folder,
                          output_folder=output_folder, fast_mode=fast_mode,
                          log=log, results=results)
@@ -21,7 +21,7 @@ class Processes(AndroidExtraction):
     def run(self):
         self._adb_connect()
 
-        output = self._adb_command("ps")
+        output = self._adb_command("ps -e")
 
         for line in output.split("\n")[1:]:
             line = line.strip()
@@ -29,13 +29,13 @@ class Processes(AndroidExtraction):
                 continue
 
             fields = line.split()
-            proc = dict(
-                user=fields[0],
-                pid=fields[1],
-                parent_pid=fields[2],
-                vsize=fields[3],
-                rss=fields[4],
-            )
+            proc = {
+                "user": fields[0],
+                "pid": fields[1],
+                "parent_pid": fields[2],
+                "vsize": fields[3],
+                "rss": fields[4],
+            }
 
             # Sometimes WCHAN is empty, so we need to re-align output fields.
             if len(fields) == 8:

mvt/android/modules/adb/rootbinaries.py

@@ -1,7 +1,7 @@
 # Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021 MVT Project Developers.
-# See the file 'LICENSE' for usage and copying permissions, or find a copy at
-#   https://github.com/mvt-project/mvt/blob/main/LICENSE
+# Copyright (c) 2021 The MVT Project Authors.
+# Use of this software is governed by the MVT License 1.1 that can be found at
+#   https://license.mvt.re/1.1/
 
 import logging
 import os
@@ -16,7 +16,7 @@ class RootBinaries(AndroidExtraction):
     """This module extracts the list of installed packages."""
 
     def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 fast_mode=False, log=None, results=[]):
+                 serial=None, fast_mode=False, log=None, results=[]):
         super().__init__(file_path=file_path, base_folder=base_folder,
                          output_folder=output_folder, fast_mode=fast_mode,
                          log=log, results=results)

mvt/android/modules/adb/sms.py

@@ -1,7 +1,7 @@
 # Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021 MVT Project Developers.
-# See the file 'LICENSE' for usage and copying permissions, or find a copy at
-#   https://github.com/mvt-project/mvt/blob/main/LICENSE
+# Copyright (c) 2021 The MVT Project Authors.
+# Use of this software is governed by the MVT License 1.1 that can be found at
+#   https://license.mvt.re/1.1/
 
 import logging
 import os
@@ -43,7 +43,7 @@ class SMS(AndroidExtraction):
     """This module extracts all SMS messages containing links."""
 
     def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 fast_mode=False, log=None, results=[]):
+                 serial=None, fast_mode=False, log=None, results=[]):
         super().__init__(file_path=file_path, base_folder=base_folder,
                          output_folder=output_folder, fast_mode=fast_mode,
                          log=log, results=results)
@@ -84,7 +84,7 @@ class SMS(AndroidExtraction):
         names = [description[0] for description in cur.description]
 
         for item in cur:
-            message = dict()
+            message = {}
             for index, value in enumerate(item):
                 message[names[index]] = value
 
@@ -102,15 +102,11 @@ class SMS(AndroidExtraction):
         log.info("Extracted a total of %d SMS messages containing links", len(self.results))
 
     def run(self):
-        # Checking the SMS database path
-        try:
-            if (self._adb_check_file_exists(os.path.join("/", SMS_BUGLE_PATH))):
-                self.SMS_DB_TYPE = 1
-                self._adb_process_file(os.path.join("/", SMS_BUGLE_PATH), self._parse_db)
-            elif (self._adb_check_file_exists(os.path.join("/", SMS_MMSSMS_PATH))):
-                self.SMS_DB_TYPE = 2
-                self._adb_process_file(os.path.join("/", SMS_MMSSMS_PATH), self._parse_db)
-            else:
-                self.log.error("No SMS database found")
-        except Exception as e:
-            self.log.error(e)
+        if (self._adb_check_file_exists(os.path.join("/", SMS_BUGLE_PATH))):
+            self.SMS_DB_TYPE = 1
+            self._adb_process_file(os.path.join("/", SMS_BUGLE_PATH), self._parse_db)
+        elif (self._adb_check_file_exists(os.path.join("/", SMS_MMSSMS_PATH))):
+            self.SMS_DB_TYPE = 2
+            self._adb_process_file(os.path.join("/", SMS_MMSSMS_PATH), self._parse_db)
+        else:
+            self.log.error("No SMS database found")

mvt/android/modules/adb/whatsapp.py

@@ -1,7 +1,7 @@
 # Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021 MVT Project Developers.
-# See the file 'LICENSE' for usage and copying permissions, or find a copy at
-#   https://github.com/mvt-project/mvt/blob/main/LICENSE
+# Copyright (c) 2021 The MVT Project Authors.
+# Use of this software is governed by the MVT License 1.1 that can be found at
+#   https://license.mvt.re/1.1/
 
 import base64
 import logging
@@ -20,7 +20,7 @@ class Whatsapp(AndroidExtraction):
     """This module extracts all WhatsApp messages containing links."""
 
     def __init__(self, file_path=None, base_folder=None, output_folder=None,
-                 fast_mode=False, log=None, results=[]):
+                 serial=None, fast_mode=False, log=None, results=[]):
         super().__init__(file_path=file_path, base_folder=base_folder,
                          output_folder=output_folder, fast_mode=fast_mode,
                          log=log, results=results)
@@ -59,7 +59,7 @@ class Whatsapp(AndroidExtraction):
 
         messages = []
         for item in cur:
-            message = dict()
+            message = {}
             for index, value in enumerate(item):
                 message[names[index]] = value
 
@@ -82,7 +82,4 @@ class Whatsapp(AndroidExtraction):
         self.results = messages
 
     def run(self):
-        try:
-            self._adb_process_file(os.path.join("/", WHATSAPP_PATH), self._parse_db)
-        except Exception as e:
-            self.log.error(e)
+        self._adb_process_file(os.path.join("/", WHATSAPP_PATH), self._parse_db)

mvt/android/modules/backup/__init__.py

@@ -1,7 +1,7 @@
 # Mobile Verification Toolkit (MVT)
-# Copyright (c) 2021 MVT Project Developers.
-# See the file 'LICENSE' for usage and copying permissions, or find a copy at
-#   https://github.com/mvt-project/mvt/blob/main/LICENSE
+# Copyright (c) 2021 The MVT Project Authors.
+# Use of this software is governed by the MVT License 1.1 that can be found at
+#   https://license.mvt.re/1.1/
 
 from .sms import SMS