Adds a custom JSON encoder class to fix serialisation issues where modules included bytes types containing non-utf8 bytes, which can't be serialised to JSON.
---------
Co-authored-by: Rory Flynn <rory.flynn@amnesty.org>
Without this change the package doesn't get properly reset when a new
user starts.
See for example in this excerpt:
```
1 | Package com.android.bluetooth:
2 | READ_CONTACTS (allow):
3 | null=[
4 | Access: [pers-s] 2022-04-22 13:24:17.577 (-277d5h22m53s447ms)
5 | ]
6 | WAKE_LOCK (allow):
7 | null=[
8 | Access: [pers-s] 2023-01-24 17:45:49.712 (-1m21s312ms) duration=+3ms
9 | ]
10 | GET_USAGE_STATS (default):
11 | null=[
12 | Reject: [pers-s]2022-04-22 13:23:53.964 (-277d5h23m17s60ms)
13 | ]
14 | BLUETOOTH_CONNECT (allow):
15 | null=[
16 | Access: [pers-s] 2022-04-22 13:23:53.988 (-277d5h23m17s36ms)
17 | ]
18 | Uid 1027:
19 | state=pers
20 | capability=LCMN
21 | appWidgetVisible=false
22 | LEGACY_STORAGE: mode=ignore
23 | Package com.android.nfc:
24 | WAKE_LOCK (allow):
25 | null=[
26 | Access: [pers-s] 2022-04-22 13:23:54.633 (-277d5h23m16s391ms) duration=+1s73ms
27 | ]
```
Here the package "com.android.bluetooth" is not reset when in line 18,
so when "LEGACY_STORAGE:" in line 22 is encountered, it's added as
another permission to "com.android.bluetooth" with "access" set to
"ode=igno".
This PR fixes that by resetting the package whenever a new Uid is
encountered.
Co-authored-by: Niclas Schwarzlose <niclas.schwarzlose@reporter-ohne-grenzen.de>
Some MVT modules such as the WhatsApp module can be very slow as it was taking a naive approach to look for IOCs. The code was checking URLs (potentially more than 100k) against
1000's of IOC domains resulting in a quadratic run-time with hundreds of millions of comparisons as the number of IOCs increases.
This commit add an Aho-Corasick library which allows the efficient search in a string (the URL in this case) for all matches in set of keys (the IOCs). This data structure is perfect for this use case.
A quick measurement shows a 80% performance improvement for a WhatsApp database with 100k entries. The slow path is now the time spent fetching and expanding short URLs found in the database. This
can also be sped up significantly by fetching each URL asynchronously. This would require reworking modules to split the URL expansion from the IOC check so I will implement in a separate PR.
* Added Documentation for Encrypted Iphone Backup with Finder on MacOS
* Added details on where to check the backups after completion and added screenshots for the process
* Added location of backups
Adds the pip comman for updating mvt. I think this would be helpfull for novice users as it already has been asked here: https://github.com/mvt-project/mvt/discussions/261
Also I sometimes forget the command too ;)
Had to run an additional Docker flag to get it building on the new M1 chip from Apple. Figured it would be helpful to point that out in the Docs for the less initiated users.
Not every system uses 'utf-8' as a default encoding for opening files in Python.
Before you say that there must be a way to set default encoding in one line, no, there is not. At least, I didn't found a way to do this.
Squashed commit of the following:
commit c0d9e8d5d188c13e7e5ec0612e99bfb7e25f47d4
Author: Donncha Ó Cearbhaill <donncha.ocearbhaill@amnesty.org>
Date: Fri Jan 7 16:05:12 2022 +0100
Update name of indicators JSON file
commit f719e49c5f942cef64931ecf422b6a6e7b8c9f17
Author: Donncha Ó Cearbhaill <donncha.ocearbhaill@amnesty.org>
Date: Fri Jan 7 15:38:03 2022 +0100
Do not set indicators option on module if no indicators were loaded
commit a289eb8de936f7d74c6c787cbb8daf5c5bec015c
Author: Donncha Ó Cearbhaill <donncha.ocearbhaill@amnesty.org>
Date: Fri Jan 7 14:43:00 2022 +0100
Simplify code for loading IoCs
commit 0804563415ee80d76c13d3b38ffe639fa14caa14
Author: Donncha Ó Cearbhaill <donncha.ocearbhaill@amnesty.org>
Date: Fri Jan 7 13:43:47 2022 +0100
Add metadata to IoC entries
commit 97d0e893c1a0736c4931363ff40f09a030b90cf6
Author: tek <tek@randhome.io>
Date: Fri Dec 17 16:43:09 2021 +0100
Implements automated loading of indicators
commit c381e14df92ae4d7d846a1c97bcf6639cc526082
Author: tek <tek@randhome.io>
Date: Fri Dec 17 12:41:15 2021 +0100
Improves download-indicators
commit b938e02ddfd0b916fd883f510b467491a4a84e5f
Author: tek <tek@randhome.io>
Date: Fri Dec 17 01:44:26 2021 +0100
Adds download-indicators for mvt-ios and mvt-android
Thank you for your interest in contributing to Mobile Verification Toolkit (MVT)! Your help is very much appreciated.
## Where to start
Starting to contribute to a somewhat complex project like MVT might seem intimidating. Unless you have specific ideas of new functionality you would like to submit, some good starting points are searching for `TODO:` and `FIXME:` comments throughout the code. Alternatively you can check if any GitHub issues existed marked with the ["help wanted"](https://github.com/mvt-project/mvt/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22) tag.
## Code style
When contributing code to
- **Indentation**: we use 4-spaces tabs.
- **Quotes**: we use double quotes (`"`) as a default. Single quotes (`'`) can be favored with nested strings instead of escaping (`\"`), or when using f-formatting.
- **Maximum line length**: we strongly encourage to respect a 80 characters long lines and to follow [PEP8 indentation guidelines](https://peps.python.org/pep-0008/#indentation) when having to wrap. However, if breaking at 80 is not possible or is detrimental to the readability of the code, exceptions are tolerated. For example, long log lines, or long strings can be extended to 100 characters long. Please hard wrap anything beyond 100 characters.
Mobile Verification Toolkit (MVT) is a collection of utilities to simplify and automate the process of gathering forensic traces helpful to identify a potential compromise of Android and iOS devices.
It has been developed and released by the [Amnesty International Security Lab](https://www.amnesty.org/en/tech/) in July 2021 in the context of the [Pegasus project](https://forbiddenstories.org/about-the-pegasus-project/) along with [a technical forensic methodology and forensic evidence](https://www.amnesty.org/en/latest/research/2021/07/forensic-methodology-report-how-to-catch-nso-groups-pegasus/).
It has been developed and released by the [Amnesty International Security Lab](https://securitylab.amnesty.org) in July 2021 in the context of the [Pegasus Project](https://forbiddenstories.org/about-the-pegasus-project/) along with [a technical forensic methodology](https://www.amnesty.org/en/latest/research/2021/07/forensic-methodology-report-how-to-catch-nso-groups-pegasus/). It continues to be maintained by Amnesty International and other contributors.
*Warning*: MVT is a forensic research tool intended for technologists and investigators. Using it requires understanding the basics of forensic analysis and using command-line tools. This is not intended for end-user self-assessment. If you are concerned with the security of your device please seek expert assistance.
> **Note**
> MVT is a forensic research tool intended for technologists and investigators. It requires understanding digital forensics and using command-line tools. This is not intended for end-user self-assessment. If you are concerned with the security of your device please seek reputable expert assistance.
>
### Indicators of Compromise
MVT supports using public [indicators of compromise (IOCs)](https://github.com/mvt-project/mvt-indicators) to scan mobile devices for potential traces of targeting or infection by known spyware campaigns. This includes IOCs published by [Amnesty International](https://github.com/AmnestyTech/investigations/) and other research groups.
> **Warning**
> Public indicators of compromise are insufficient to determine that a device is "clean", and not targeted with a particular spyware tool. Reliance on public indicators alone can miss recent forensic traces and give a false sense of security.
>
> Reliable and comprehensive digital forensic support and triage requires access to non-public indicators, research and threat intelligence.
>
>Such support is available to civil society through [Amnesty International's Security Lab](https://www.amnesty.org/en/tech/) or through our forensic partnership with [Access Now’s Digital Security Helpline](https://www.accessnow.org/help/).
More information about using indicators of compromise with MVT is available in the [documentation](https://docs.mvt.re/en/latest/iocs/).
## Installation
MVT can be installed from sources or from [PyPi](https://pypi.org/project/mvt/) (you will need some dependencies, check the [documentation](https://docs.mvt.re/en/latest/install.html)):
MVT can be installed from sources or from [PyPI](https://pypi.org/project/mvt/) (you will need some dependencies, check the [documentation](https://docs.mvt.re/en/latest/install/)):
```
pip3 install mvt
```
Alternatively, you can decide to run MVT and all relevant tools through a [Docker container](https://docs.mvt.re/en/latest/docker.html).
For alternative installation options and known issues, please refer to the [documentation](https://docs.mvt.re/en/latest/install/) as well as [GitHub Issues](https://github.com/mvt-project/mvt/issues).
**Please note:** MVT is best run on Linux or Mac systems. [It does not currently support running natively on Windows.](https://docs.mvt.re/en/latest/install.html#mvt-on-windows)
## Usage
MVT provides two commands `mvt-ios` and `mvt-android`. [Check out the documentation to learn how to use them!](https://docs.mvt.re/)
## License
The purpose of MVT is to facilitate the ***consensual forensic analysis*** of devices of those who might be targets of sophisticated mobile spyware attacks, especially members of civil society and marginalized communities. We do not want MVT to enable privacy violations of non-consenting individuals. In order to achieve this, MVT is released under its own license. [Read more here.](https://docs.mvt.re/en/latest/license.html)
The purpose of MVT is to facilitate the ***consensual forensic analysis*** of devices of those who might be targets of sophisticated mobile spyware attacks, especially members of civil society and marginalized communities. We do not want MVT to enable privacy violations of non-consenting individuals. In order to achieve this, MVT is released under its own license. [Read more here.](https://docs.mvt.re/en/latest/license/)
Thank you for your interest in reporting security issues and vulnerabilities! Security research is of utmost importance and we take all reports seriously. If you discover an issue please report it to us right away!
Please DO NOT file a public issue, instead send your report privately to *nex [at] nex [dot] sx*. You can also write PGP-encrypted emails to [this key](https://keybase.io/nex/pgp_keys.asc?fingerprint=05216f3b86848a303c2fe37dd166f1667359d880).
In case you nonetheless wish to take a full backup, you can do so with
```bash
adb backup -all
adb backup -nocompress -all
```
## Unpack the backup
Some recent phones will enforce the utilisation of a password to encrypt the backup archive. In that case, the password will obviously be needed to extract and analyse the data later on.
In order to reliable unpack th [Android Backup Extractor (ABE)](https://github.com/nelenkov/android-backup-extractor) to convert it to a readable file format. Make sure that java is installed on your system and use the following command:
## Unpack and check the backup
MVT includes a partial implementation of the Android Backup parsing, because of the implementation difference in the compression algorithm between Java and Python. The `-nocompress` option passed to adb in the section above allows to avoid this issue. You can analyse and extract SMSs from the backup directly with MVT:
INFO [mvt.android.modules.backup.sms] Extracted a total of 64 SMS messages
```
If the backup is encrypted, MVT will prompt you to enter the password. A backup password can also be provided with the `--backup-password` command line option or through the `MVT_ANDROID_BACKUP_PASSWORD` environment variable. The same options can also be used to when analysing an encrypted backup collected through AndroidQF in the `mvt-android check-androidqf` command:
Through the `--iocs` argument you can specify a [STIX2](https://oasis-open.github.io/cti-documentation/stix/intro) file defining a list of malicious indicators to check against the records extracted from the backup by MVT. Any matches will be highlighted in the terminal output.
## Alternative ways to unpack and check the backup
If you encounter an issue during the analysis of the backup, you can alternatively use [Android Backup Extractor (ABE)](https://github.com/nelenkov/android-backup-extractor) to convert it to a readable file format. Make sure that java is installed on your system and use the following command:
16:18:38 INFO [mvt.android.cli] Checking ADB backup located at: .
INFO [mvt.android.modules.backup.sms] Running module SMS...
INFO [mvt.android.modules.backup.sms] Processing SMS backup file at /path/to/backup/apps/com.android.providers.telephony/d_f/000000_sms_backup
16:18:39 INFO [mvt.android.modules.backup.sms] Extracted a total of
64 SMS messages containing links
```
Through the `--iocs` argument you can specify a [STIX2](https://oasis-open.github.io/cti-documentation/stix/intro) file defining a list of malicious indicators to check against the records extracted from the backup by MVT. Any matches will be highlighted in the terminal output.
You can then extract SMSs with MVT by passing the folder path as parameter instead of the `.ab` file: `mvt-android check-backup --output /path/to/results/ /path/to/backup/` (the path to backup given should be the folder containing the `apps` folder).
@@ -13,22 +13,16 @@ It might take several minutes to complete.
!!! info
MVT will likely warn you it was unable to download certain installed packages. There is no reason to be alarmed: this is typically expected behavior when MVT attempts to download a system package it has no privileges to access.
Optionally, you can decide to enable lookups of the SHA256 hash of all the extracted APKs on [VirusTotal](https://www.virustotal.com) and/or [Koodous](https://koodous.com). While these lookups do not provide any conclusive assessment on all of the extracted APKs, they might highlight any known malicious ones:
Optionally, you can decide to enable lookups of the SHA256 hash of all the extracted APKs on [VirusTotal](https://www.virustotal.com). While these lookups do not provide any conclusive assessment on all of the extracted APKs, they might highlight any known malicious ones:
Please note that in order to use VirusTotal lookups you are required to provide your own API key through the `MVT_VT_API_KEY` environment variable. You should also note that VirusTotal enforces strict API usage. Be mindful that MVT might consume your hourly search quota.
In case you have a previous extraction of APKs you want to later check against VirusTotal, you can do so with the following arguments:
Because malware attacks over Android typically take the form of malicious or backdoored apps, the very first thing you might want to do is to extract and verify all installed Android packages and triage quickly if there are any which stand out as malicious or which might be atypical.
While it is out of the scope of this documentation to dwell into details on how to analyze Android apps, MVT does allow to easily and automatically extract information about installed apps, download copies of them, and quickly lookup services such as [VirusTotal](https://www.virustotal.com) or [Koodous](https://koodous.com) which might quickly indicate known bad apps.
While it is out of the scope of this documentation to dwell into details on how to analyze Android apps, MVT does allow to easily and automatically extract information about installed apps, download copies of them, and quickly look them up on services such as [VirusTotal](https://www.virustotal.com).
!!! info "Using VirusTotal"
Please note that in order to use VirusTotal lookups you are required to provide your own API key through the `MVT_VT_API_KEY` environment variable. You should also note that VirusTotal enforces strict API usage. Be mindful that MVT might consume your hourly search quota.
The Mobile Verification Toolkit team welcomes contributions of new forensic modules or other contributions which help improve the software.
## Testing
MVT uses `pytest` for unit and integration tests. Code style consistency is maintained with `flake8`, `ruff` and `black`. All can
be run automatically with:
```bash
make check
```
Run these tests before making new commits or opening pull requests.
## Profiling
Some MVT modules extract and process significant amounts of data during the analysis process or while checking results against known indicators. Care must be
take to avoid inefficient code paths as we add new modules.
MVT modules can be profiled with Python built-in `cProfile` by setting the `MVT_PROFILE` environment variable.
Using Docker simplifies having all the required dependencies and tools (including most recent versions of [libimobiledevice](https://libimobiledevice.org)) readily installed.
Using Docker simplifies having all the required dependencies and tools (including most recent versions of [libimobiledevice](https://libimobiledevice.org)) readily installed. Note that this requires a Linux host, as Docker for Windows and Mac [doesn't support passing through USB devices](https://docs.docker.com/desktop/faqs/#can-i-pass-through-a-usb-device-to-a-container).
Install Docker following the [official documentation](https://docs.docker.com/get-docker/).
Mobile Verification Toolkit (MVT) is a tool to facilitate the [consensual forensic analysis](introduction.md#consensual-forensics) of Android and iOS devices, for the purpose of identifying traces of compromise.
It has been developed and released by the [Amnesty International Security Lab](https://securitylab.amnesty.org) in July 2021 in the context of the [Pegasus Project](https://forbiddenstories.org/about-the-pegasus-project/) along with [a technical forensic methodology](https://www.amnesty.org/en/latest/research/2021/07/forensic-methodology-report-how-to-catch-nso-groups-pegasus/). It continues to be maintained by Amnesty International and other contributors.
In this documentation you will find instructions on how to install and run the `mvt-ios` and `mvt-android` commands, and guidance on how to interpret the extracted results.
@@ -42,19 +42,19 @@ It is recommended to try installing and running MVT from [Windows Subsystem Linu
## Installing MVT
If you haven't done so, you can add this to your `.bashrc` or `.zshrc` file in order to add locally installed Pypi binaries to your `$PATH`:
If you haven't done so, you can add this to your `.bashrc` or `.zshrc` file in order to add locally installed PyPI binaries to your `$PATH`:
```bash
export PATH=$PATH:~/.local/bin
```
Then you can install MVT directly from [pypi](https://pypi.org/project/mvt/)
Then you can install MVT directly from [PyPI](https://pypi.org/project/mvt/)
```bash
pip3 install mvt
```
Or from the source code:
If you want to have the latest features in development, you can install MVT directly from the source code. If you installed MVT previously from pypi, you should first uninstall it using `pip3 uninstall mvt` and then install from the source code:
@@ -12,6 +12,20 @@ Mobile Verification Toolkit (MVT) is a collection of utilities designed to facil
MVT is a forensic research tool intended for technologists and investigators. Using it requires understanding the basics of forensic analysis and using command-line tools. MVT is not intended for end-user self-assessment. If you are concerned with the security of your device please seek expert assistance.
## Indicators of Compromise
MVT supports using [indicators of compromise (IOCs)](https://github.com/mvt-project/mvt-indicators) to scan mobile devices for potential traces of targeting or infection by known spyware campaigns. This includes IOCs published by [Amnesty International](https://github.com/AmnestyTech/investigations/) and other research groups.
!!! warning
Public indicators of compromise are insufficient to determine that a device is "clean", and not targeted with a particular spyware tool. Reliance on public indicators alone can miss recent forensic traces and give a false sense of security.
Reliable and comprehensive digital forensic support and triage requires access to non-public indicators, research and threat intelligence.
Such support is available to civil society through [Amnesty International's Security Lab](https://securitylab.amnesty.org/contact-us/) or [Access Now’s Digital Security Helpline](https://www.accessnow.org/help/).
More information about using indicators of compromise with MVT is available in the [documentation](iocs.md).
## Consensual Forensics
While MVT is capable of extracting and processing various types of very personal records typically found on a mobile phone (such as calls history, SMS and WhatsApp messages, etc.), this is intended to help identify potential attack vectors such as malicious SMS messages leading to exploitation.
- [Predator from Cytrox](https://citizenlab.ca/2021/12/pegasus-vs-predator-dissidents-doubly-infected-iphone-reveals-cytrox-mercenary-spyware/) ([STIX2](https://raw.githubusercontent.com/AmnestyTech/investigations/master/2021-12-16_cytrox/cytrox.stix2))
- [An Android Spyware Campaign Linked to a Mercenary Company](https://github.com/AmnestyTech/investigations/tree/master/2023-03-29_android_campaign) ([STIX2](https://github.com/AmnestyTech/investigations/blob/master/2023-03-29_android_campaign/malware.stix2))
- [This repository](https://github.com/Te-k/stalkerware-indicators) contains IOCs for Android stalkerware including [a STIX MVT-compatible file](https://raw.githubusercontent.com/Te-k/stalkerware-indicators/master/generated/stalkerware.stix2).
- We are also maintaining [a list of IOCs](https://github.com/mvt-project/mvt-indicators) in STIX format from public spyware campaigns.
You can automaticallly download the latest public indicator files with the command `mvt-ios download-iocs` or `mvt-android download-iocs`. These commands download the list of indicators from the [mvt-indicators](https://github.com/mvt-project/mvt-indicators/blob/main/indicators.yaml) repository and store them in the [appdir](https://pypi.org/project/appdirs/) folder. They are then loaded automatically by MVT.
Please [open an issue](https://github.com/mvt-project/mvt/issues/) to suggest new sources of STIX-formatted IOCs.
It is possible to do an iPhone backup by using iTunes on Windows or macOS computers (in most recent versions of macOS, this feature is included in Finder).
It is possible to do an iPhone backup by using iTunes on Windows or macOS computers (in most recent versions of macOS, this feature is included in Finder, see below).
To do that:
* Make sure iTunes is installed.
* Connect your iPhone to your computer using a Lightning/USB cable.
* Open the device in iTunes (or Finder on macOS).
* If you want to have a more accurate detection, ensure that the encrypted backup option is activated and choose a secure password for the backup.
* Start the backup and wait for it to finish (this may take up to 30 minutes).
1. Make sure iTunes is installed.
2. Connect your iPhone to your computer using a Lightning/USB cable.
3. Open the device in iTunes (or Finder on macOS).
4. If you want to have a more accurate detection, ensure that the encrypted backup option is activated and choose a secure password for the backup.
5. Start the backup and wait for it to finish (this may take up to 30 minutes).
* Once the backup is done, find its location and copy it to a place where it can be analyzed by MVT. On Windows, the backup can be stored either in `%USERPROFILE%\Apple\MobileSync\` or `%USERPROFILE%\AppData\Roaming\Apple Computer\MobileSync\`. On macOS, the backup is stored in `~/Library/Application Support/MobileSync/`.
Once the backup is done, find its location and copy it to a place where it can be analyzed by MVT. On Windows, the backup can be stored either in `%USERPROFILE%\Apple\MobileSync\` or `%USERPROFILE%\AppData\Roaming\Apple Computer\MobileSync\`. On macOS, the backup is stored in `~/Library/Application Support/MobileSync/`.
# Backup with Finder
On more recent MacOS versions, this feature is included in Finder. To do a backup:
1. Launch Finder on your Mac.
2. Connect your iPhone to your Mac using a Lightning/USB cable.
3. Select your device from the list of devices located at the bottom of the left side bar labeled "locations".
4. In the General tab, select `Back up all the data on your iPhone to this Mac` from the options under the Backups section.
5. Check the box that says `Encrypt local backup`. If it is your first time selecting this option, you may need to enter a password to encrypt the backup.
6. Click `Back Up Now` to start the back-up process.
7. The encrypted backup for your iPhone should now start. Once the process finishes, you can check the backup by opening `Finder`, clicking on the `General` tab, then click on `Manage Backups`. Now you should see a list of your backups like the image below:
If your backup has a lock next to it like in the image above, then the backup is encrypted. You should also see the date and time when the encrypted backup was created. The backup files are stored in `~/Library/Application Support/MobileSync/`.
## Notes:
- Remember to keep the backup encryption password that you created safe, since without it you will not be able to access/modify/decrypt the backup file.
If you have correctly [installed libimobiledevice](../install.md) you can easily generate an iTunes backup using the `idevicebackup2` tool included in the suite. First, you might want to ensure that backup encryption is enabled (**note: encrypted backup contain more data than unencrypted backups**):
```bash
idevicebackup2 -i backup encryption on
idevicebackup2 -i encryption on
```
Note that if a backup password was previously set on this device, you might need to use the same or change it. You can try changing password using `idevicebackup2 -i backup changepw`, or by turning off encryption (`idevicebackup2 -i backup encryption off`) and turning it back on again.
Note that if a backup password was previously set on this device, you might need to use the same or change it. You can try changing password using `idevicebackup2 -i changepw`, or by turning off encryption (`idevicebackup2 -i encryption off`) and turning it back on again.
If you are not able to recover or change the password, you should try to disable encryption and obtain an unencrypted backup.
# Use of this software is governed by the MVT License 1.1 that can be found at
# https://license.mvt.re/1.1/
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
