1
mirror of https://github.com/home-assistant/core synced 2024-09-18 19:55:20 +02:00
Go to file
Greg Laabs 6e1785173f History query and schema optimizations for huge performance boost (#8748)
* Add DEBUG-level log for db row to native object conversion

This is now the bottleneck (by a large margin) for big history queries, so I'm leaving this log feature in to help diagnose users with a slow history page

* Rewrite of the "first synthetic datapoint" query for multiple entities

The old method was written in a manner that prevented an index from being used in the inner-most GROUP BY statement, causing massive performance issues especially when querying for a large time period.

The new query does have one material change that will cause it to return different results than before: instead of using max(state_id) to get the latest entry, we now get the max(last_updated). This is more appropriate (primary key should not be assumed to be in order of event firing) and allows an index to be used on the inner-most query. I added another JOIN layer to account for cases where there are two entries on the exact same `last_created` for a given entity. In this case we do use `state_id` as a tiebreaker.

For performance reasons the domain filters were also moved to the outermost query, as it's way more efficient to do it there than on the innermost query as before (due to indexing with GROUP BY problems)

The result is a query that only needs to do a filesort on the final result set, which will only be as many rows as there are entities.

* Remove the ORDER BY entity_id when fetching states, and add logging

Having this ORDER BY in the query prevents it from using an index due to the range filter, so it has been removed.

We already do a `groupby` in the `states_to_json` method which accomplishes exactly what the ORDER BY in the query was trying to do anyway, so this change causes no functional difference.

Also added DEBUG-level logging to allow diagnosing a user's slow history page.

* Add DEBUG-level logging for the synthetic-first-datapoint query

For diagnosing a user's slow history page

* Missed a couple instances of `created` that should be `last_updated`

* Remove `entity_id` sorting from state_changes; match significant_update

This is the same change as 09b3498f41 , but applied to the `state_changes_during_period` method which I missed before. This should give the same performance boost to the history sensor component!

* Bugfix in History query used for History Sensor

The date filter was using a different column for the upper and lower bounds. It would work, but it would be slow!

* Update Recorder purge script to use more appropriate columns

Two reasons: 1. the `created` column's meaning is fairly arbitrary and does not represent when an event or state change actually ocurred. It seems more correct to purge based on the event date than the time the database row was written.
2. The new columns are indexed, which will speed up this purge script by orders of magnitude

* Updating db model to match new query optimizations

A few things here: 1. New schema version with a new index and several removed indexes
2. A new method in the migration script to drop old indexes
3. Added an INFO-level log message when a new index will be added, as this can take quite some time on a Raspberry Pi
2017-08-04 23:16:53 -07:00
.github Use H2 headers to split up the different sections (#6183) 2017-02-23 12:58:18 -08:00
docs Update links.html 2017-05-24 14:47:22 -07:00
homeassistant History query and schema optimizations for huge performance boost (#8748) 2017-08-04 23:16:53 -07:00
script Block dependencies that depend on enum34 (#8698) 2017-08-04 23:06:10 -07:00
tests History query and schema optimizations for huge performance boost (#8748) 2017-08-04 23:16:53 -07:00
virtualization Block dependencies that depend on enum34 (#8698) 2017-08-04 23:06:10 -07:00
.coveragerc Xiaomi vacuum as platform of new vacuum component derived from ToggleEntity, and services (#8623) 2017-08-04 15:27:10 +02:00
.dockerignore Docker cleanup. (#8226) 2017-06-30 08:07:33 -07:00
.gitignore Move examples out (#6908) 2017-04-02 17:01:51 -07:00
.gitmodules Update links 2016-04-12 14:18:18 +02:00
.hound.yml Add .hound.yml 2016-11-13 11:34:09 -08:00
.ignore Include .ignore file for search utilities (#5290) 2017-01-16 22:52:53 -08:00
.travis.yml Disable Python 3.6-dev while it's broken 2017-07-02 11:52:24 -07:00
CLA.md Update LICENSE.md and CLA.md to reflect the new Apache 2.0 license 2017-01-28 12:12:34 -08:00
CODE_OF_CONDUCT.md Add organization docs 2017-01-22 16:21:20 -08:00
CODEOWNERS Fix CODEOWNERS z-wave team name (#8400) 2017-07-07 17:00:14 -07:00
CONTRIBUTING.md Fix link to pull request advice for contributors 2017-01-03 14:26:24 +11:00
Dockerfile Block dependencies that depend on enum34 (#8698) 2017-08-04 23:06:10 -07:00
LICENSE.md Update LICENSE.md and CLA.md to reflect the new Apache 2.0 license 2017-01-28 12:12:34 -08:00
MANIFEST.in [packaging] Include LICENSE.md in tarball (#6514) 2017-03-10 19:59:38 +00:00
pylintrc Add support for using credstash as a secret store (#8494) 2017-07-24 08:59:10 -07:00
README.rst Update README.rst 2017-07-24 07:53:14 -07:00
requirements_all.txt Upgrade aiohttp to 2.2.5 (#8828) 2017-08-04 23:14:05 -07:00
requirements_docs.txt Upgrade sphinx-autodoc-typehints to 1.2.1 (#8783) 2017-08-02 14:15:00 +02:00
requirements_test_all.txt Fix Dyson sensors if devices are configured without standby monitoring. Fixes #8569 (#8826) 2017-08-04 14:27:23 -07:00
requirements_test.txt Upgrade mypy to 0.521 (#8692) 2017-07-28 23:22:56 -07:00
setup.cfg Expose isort preferences for tools. (#4481) 2016-11-27 14:33:30 -08:00
setup.py Upgrade aiohttp to 2.2.5 (#8828) 2017-08-04 23:14:05 -07:00
tox.ini Block dependencies that depend on enum34 (#8698) 2017-08-04 23:06:10 -07:00

Home Assistant |Build Status| |Coverage Status| |Chat Status|
=============================================================

Home Assistant is a home automation platform running on Python 3. It is able to track and control all devices at home and offer a platform for automating control.

To get started:

.. code:: bash

    python3 -m pip install homeassistant
    hass --open-ui

Check out `home-assistant.io <https://home-assistant.io>`__ for `a
demo <https://home-assistant.io/demo/>`__, `installation instructions <https://home-assistant.io/getting-started/>`__,
`tutorials <https://home-assistant.io/getting-started/automation-2/>`__ and `documentation <https://home-assistant.io/docs/>`__.

|screenshot-states|

Featured integrations
---------------------

|screenshot-components|

The system is built using a modular approach so support for other devices or actions can be implemented easily. See also the `section on architecture <https://home-assistant.io/developers/architecture/>`__ and the `section on creating your own
components <https://home-assistant.io/developers/creating_components/>`__.

If you run into issues while using Home Assistant or during development
of a component, check the `Home Assistant help section <https://home-assistant.io/help/>`__ of our website for further help and information.

.. |Build Status| image:: https://travis-ci.org/home-assistant/home-assistant.svg?branch=master
   :target: https://travis-ci.org/home-assistant/home-assistant
.. |Coverage Status| image:: https://img.shields.io/coveralls/home-assistant/home-assistant.svg
   :target: https://coveralls.io/r/home-assistant/home-assistant?branch=master
.. |Chat Status| image:: https://img.shields.io/discord/330944238910963714.svg
   :target: https://discord.gg/c5DvZ4e
.. |Join the chat at https://gitter.im/home-assistant/home-assistant| image:: https://img.shields.io/badge/gitter-general-blue.svg
   :target: https://gitter.im/home-assistant/home-assistant?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge
.. |Join the dev chat at https://gitter.im/home-assistant/home-assistant/devs| image:: https://img.shields.io/badge/gitter-development-yellowgreen.svg
   :target: https://gitter.im/home-assistant/home-assistant/devs?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge
.. |screenshot-states| image:: https://raw.github.com/home-assistant/home-assistant/master/docs/screenshots.png
   :target: https://home-assistant.io/demo/
.. |screenshot-components| image:: https://raw.github.com/home-assistant/home-assistant/dev/docs/screenshot-components.png
   :target: https://home-assistant.io/components/