We do not need to capture HostNotSupported errors to Sentry. The only
possible code path this error might come from is where the Journal
Gateway Daemon socket is unavailable, which is already reported as an
"Unsupported system" repair.
* Use signals to recognize new disks immediately
* Add test for disabled data disk issue
* Add mock of UDisks2 base service to test
* Apply suggestions from code review
* Shutdown manager first to avoid potential race conditions
* Update tests/dbus_service_mocks/udisks2.py
Co-authored-by: Jan Čermák <sairon@users.noreply.github.com>
---------
Co-authored-by: Stefan Agner <stefan@agner.ch>
Co-authored-by: Jan Čermák <sairon@users.noreply.github.com>
Supervisor logs fallback in get_supervisor_logs didn't work properly
because the exception was caught in api_process_raw instead. This was
not discovered in tests because the side effect raised OSError, which
isn't handled there.
To address that, I split the advanced_logs to two functions, one being a
wrapped API handler, one being plain function returning response without
any additional error handling. The tests now check for both cases of
errors (HassioError and random generic Python error).
Refs #5021
* Allow adoption of existing data disk
* Fix existing tests
* Add test cases and fix image issues
* Fix addon build test
* Run checks during setup not startup
* Addon load mimics plugin and HA load for docker part
* Default image accessible in except
* Use Systemd Journal API for all logs endpoints in API
Replace all logs endpoints using container logs with wrapped
advanced_logs function, adding possibility to get logs from previous
boots and following the logs. Supervisor logs are an excetion where
Docker logs are still used - in case an exception is raised while
accessing the Systemd logs, they're used as fallback - otherwise we
wouldn't have an easy way to see what went wrong.
* Refactor testing of advanced logs endpoints to a common method
* Send error while fetching Supervisor logs to Sentry; minor cleanup
* Properly handle errors and use consistent content type in logs endpoints
* Replace api_process_custom with reworked api_process_raw per @mdegat01 suggestion
As described in #4996, Docker 25+ changes made sub-mounts of the /dev
filesystem to be mounted read-only. Revert to the previous behavior by
adjusting the ReadOnlyNonRecursive option. Cleaner way would be to
upstream support for setting this option via Mount class arguments, so
this change is meant to be rather a hotfix for the issue. Even better
approach would be mounting /dev non-recursively, and taking care of
creating all necessary filesystems when creating containers in
Supervisor.
* Unsupported if wrong image used on virtualization
* Add generic-aarch64 as supported image
* Add virtualization field to API
* Change startup to setup in check
* Use Journal Export Format for host (advanced) logs
Add methods for handling Journal Export Format and use it for fetching
of host logs. This is foundation for colored streaming logs for other
endpoints as well.
* Make pylint happier - remove extra pass statement
* Rewrite journal gateway tests to mock ClientResponse's StreamReader
* Handle connection refused error when connecting to journal-gatewayd
* Use SYSTEMD_JOURNAL_GATEWAYD_SOCKET global path also for connection
* Use parsing algorithm suggested by @agners in review
* Fix timestamps in formatting, always use UTC for now
* Add tests for Accept header in host logs
* Apply suggestions from @agners
Co-authored-by: Stefan Agner <stefan@agner.ch>
* Bail out of parsing earlier if field is not in required fields
* Fix parsing issue discovered in the wild and add test case
* Make verbose formatter more tolerant
* Use some bytes' native functions for some minor optimizations
* Move MalformedBinaryEntryError to exceptions module, add test for it
---------
Co-authored-by: Stefan Agner <stefan@agner.ch>
* Cache existence of addon paths
* Always update submodules
* Switch to an always cached model
* Cache on store addon only
* Fix tests
* refresh_cache to refresh_path_cache
* Fix name change in test
* Move logic into StoreManager
* Allow client to change boot slot via API
* Wrap call to rauc in job that checks for OS
* Reboot after changing the active boot slot
* Add test cases and clean up
* BootName to BootSlot
* Fix test
* Rename boot_name to boot_slot
* Fix tests after field change
* Bump pytest from 7.4.4 to 8.0.1
Bumps [pytest](https://github.com/pytest-dev/pytest) from 7.4.4 to 8.0.1.
- [Release notes](https://github.com/pytest-dev/pytest/releases)
- [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst)
- [Commits](https://github.com/pytest-dev/pytest/compare/7.4.4...8.0.1)
---
updated-dependencies:
- dependency-name: pytest
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
* Update pytest-asyncio to 0.23.5
* Set scope to function on fixture
* Unthrottle by patching last call to prevent carryover
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Stefan Agner <stefan@agner.ch>
Co-authored-by: Mike Degatano <michael.degatano@gmail.com>
* Migrate to Ruff for lint and format
* Fix pylint issues
* DBus property sets into normal awaitable methods
* Fix tests relying on separate tasks in connect
* Fixes from feedback
* Reset failed API call counter on successful API call
Make sure to reset the failed API call counter after a successful
API call. While at it also update the log messages a bit to make it
clearer what the problem is exactly.
* Address pytest changes
* Add background option to backup APIs
* Fix decorator tests
* Working error handling, initial test cases
* Change to schedule_job and always return job id
* Add tests
* Reorder call at/later args
* Validation errors return immediately in background
* None is invalid option for background
* Must pop the background option from body
* Improve json performance by porting core orjson utils
* port relevant tests
* pylint
* add test for read_json_file
* add test for read_json_file
* remove workaround for core issue we do not have here
---------
Co-authored-by: Pascal Vizeli <pvizeli@syshack.ch>
* Added support for setting addon target path in map config
* Updated addon target path mapping to use dataclass
* Added check before adding string folder maps
* Moved enum to addon/const, updated map_volumes logic, fixed test
* Removed log used for debugging
* Use more readable approach to determine addon_config_used
Co-authored-by: Mike Degatano <michael.degatano@gmail.com>
* Use cleaner approach for checking volume config
Co-authored-by: Mike Degatano <michael.degatano@gmail.com>
* Use dict syntax and ATTR_TYPE
Co-authored-by: Mike Degatano <michael.degatano@gmail.com>
* Use coerce for validating mapping type
Co-authored-by: Mike Degatano <michael.degatano@gmail.com>
* Default read_only to true in schema
Co-authored-by: Mike Degatano <michael.degatano@gmail.com>
* Use ATTR_TYPE and ATTR_READ_ONLY instead of static strings
Co-authored-by: Mike Degatano <michael.degatano@gmail.com>
* Use constants instead of in-line strings
Co-authored-by: Mike Degatano <michael.degatano@gmail.com>
* Correct type for path
Co-authored-by: Mike Degatano <michael.degatano@gmail.com>
* Added read_only and path constants
* Fixed small syntax error and added includes for constants
* Simplify logic for handling string and dict entries in map config
* Use ATTR_PATH instead of inline string
Co-authored-by: Mike Degatano <michael.degatano@gmail.com>
* Add missing ATTR_PATH reference
* Moved FolderMapping dataclass to data.py
* Fix edge case where "data" map type is used but optional path is not set
* Move FolderMapping dataclass to configuration.py to prevent circular reference
---------
Co-authored-by: Jeff Oakley <jeff.oakley@LearningCircleSoftware.com>
Co-authored-by: Mike Degatano <michael.degatano@gmail.com>
Co-authored-by: Pascal Vizeli <pvizeli@syshack.ch>
* Pass architecture of installed add-on on update
When using multi-architecture container images, the architecture of the
add-on is not passed to Docker in all cases. This causes the
architecture of the Supervisor container to be used, which potentially
is not supported by the add-on in question.
This commit passes the architecture of the add-on to Docker, so that
the correct image is pulled.
* Call update with architecture
* Also pass architecture on add-on restore
* Fix pytest
* Bad message error marks system as unhealthy
* Finish adding test cases for changes
* Rename test file for uniqueness
* bad_message to oserror_bad_message
* Omit some checks and check for network mounts
As shown in home-assistant/operating-system#3007, error messages printed
to logs when container installation fails can cause some confusion,
because they are sometimes printed to the log on the landing page.
Adjust all wordings of "retry in" to "retrying in" to make it obvious
this happens automatically.
* Fix fallback to non-SSL whoami call
In case of an exception "data" is not set leading to an error:
cannot access local variable 'data' where it is not associated with a value
Make sure to fallback to the non-SSL whoami call properly.
* Add pytests
* Ignore protected access in pytests
* Add test when system time is behind by more than 3 days
* Fix test_adjust_system_datetime_if_time_behind test and cleanup
Instead of setting the ingress port on install, make sure to set
the port when the add-on gets loaded (on Supervisor startup and
before installation). This is necessary since the dynamic ingress
ports are not stored as part of the add-on data storage themself
but in the ingress data store. So on every Supervisor start the
port needs to be transferred to the add-on model.
Note that we still need to check the port on add-on update since
the add-on potentially added (dynamic) ingress on update. Same
applies to add-on restore (the restored version might use a dynamic
ingress port).
* Make check_port asyncio
This requires to change the ingress_port property to a async method.
* Avoid using wait_for
* Add missing async
* Really await
* Set dynamic ingress port on add-on installation/update
* Fix pytest issue
* Rename async_check_port back to check_port
* Raise RuntimeError in case port is not set
* Make sure port gets set on add-on restore
* Drop unnecessary async
* Simplify check_port by using asyncio.get_running_loop()
* Wait until mount unit is deactivated on unmount
The current code does not wait until the (bind) mount unit has been
actually deactivated (state "inactive"). This is especially problematic
when restoring a backup, where we deactivate all bind mounts before
restoring the target folder. Before the tarball is actually restored,
we delete all contents of the target folder. This lead to the situation
where the "rm -rf" command got executed before the bind mount actually
got unmounted.
The current code polls the state using an exponentially increasing
delay. Wait up to 30s for the bind mount to actually deactivate.
* Fix function name
* Fix missing await
* Address pytest errors
Change state of systemd unit according to use cases. Note that this
is currently rather fragile, and ideally we should have a smarter
mock service instead.
* Fix pylint
* Fix remaining
* Check transition fo failed as well
* Used alternative mocking mechanism
* Remove state lists in test_manager
---------
Co-authored-by: Mike Degatano <michael.degatano@gmail.com>
* Improve D-Bus error handling for NetworkManager
There are quite some errors captured which are related by seemingly a
suddenly missing NetworkManager. Errors appear as:
23-11-21 17:42:50 ERROR (MainThread) [supervisor.dbus.network] Error while processing /org/freedesktop/NetworkManager/Devices/10: Remote peer disconnected
...
23-11-21 17:42:50 ERROR (MainThread) [supervisor.dbus.network] Error while processing /org/freedesktop/NetworkManager/Devices/35: The name is not activatable
Both errors seem to already happen at introspection time, however
the current code doesn't converts these errors to Supervisor issues.
This PR uses the already existing `DBus.from_dbus_error()`.
Furthermore this adds a new Exception `DBusNoReplyError` for the
`ErrorType.NO_REPLY` (or `org.freedesktop.DBus.Error.NoReply` in
D-Bus terms, which is the type of the first of the two issues above).
And finally it separates the `ErrorType.SERVICE_UNKNOWN` (or
`org.freedesktop.DBus.Error.ServiceUnknown` in D-Bus terms, which is
the second of the above issue) from `DBusInterfaceError` into a new
`DBusServiceUnkownError`.
This allows to handle errors more specifically.
To avoid too much churn, all instances where `DBusInterfaceError`
got handled, we are now also handling `DBusServiceUnkownError`.
The `DBusNoReplyError` and `DBusServiceUnkownError` appear when
the NetworkManager service stops or crashes. Instead of retrying
every interface we know, just give up if one of these issues appear.
This should significantly lower error messages users are seeing
and Sentry events.
* Remove unnecessary statement
* Fix pytests
* Make sure error strings are compared correctly
* Fix typo/remove unnecessary pylint exception
* Fix DBusError typing
* Add pytest for from_dbus_error
* Revert "Make sure error strings are compared correctly"
This reverts commit 10dc2e4c38.
* Add test cases
---------
Co-authored-by: Mike Degatano <michael.degatano@gmail.com>
Jan Čermák