Release 0.19.1

Hotfix for the latest database migration

.github/ISSUE_TEMPLATE.md

@@ -1,3 +0,0 @@
-- [ ] I carefully read the [contribution guidelines](https://github.com/TeamNewPipe/NewPipe/blob/HEAD/.github/CONTRIBUTING.md) and agree to them.
-- [ ] I checked if the issue/feature exists in the latest version.
-- [ ] I did use the [incredible bugreport to markdown converter](https://teamnewpipe.github.io/CrashReportToMarkdown/) to paste bug reports.

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,44 @@
+---
+name: Bug report
+about: Create a bug report to help us improve
+labels: bug
+assignees: ''
+
+---
+
+<!--
+Oh no, a bug! It happens. Thanks for reporting an issue with NewPipe.
+
+Use this template to notify us if you found a bug.
+
+To make it easier for us to help you please enter detailed information below.
+
+Please note, we only support the latest version of NewPipe and the master branch. Make sure you have that version installed. If you don't, upgrade & reproduce the problem before opening the issue. The release page (https://github.com/TeamNewPipe/NewPipe/releases/latest) is the go-to place to get this version. In order to check your app version, open the left drawer and click on "About".
+
+P.S.: Our contribution guidelines might be a nice document to read before you fill out the report :) You can find it at https://github.com/TeamNewPipe/NewPipe/blob/HEAD/.github/CONTRIBUTING.md
+-->
+### Version
+<!-- Which version are you using? -->
+- 
+
+
+### Steps to reproduce the bug
+<!-- If you can't reproduce it, please try to give as many details as possible on how you think you got to the bug. -->
+Steps to reproduce the behavior:
+1. Go to '...'
+2. Press on '....'
+3. Swipe down to '....'
+
+### Expected behavior
+Tell us what you expected to happen.
+
+### Actual behaviour
+Tell us what happens instead.
+
+### Screenshots/Screen records
+If applicable, add screenshots or a screen recording to help explain your problem. GitHub should support uploading them directly in the issue field. If your file is too big, feel free to paste a link from an image/video hoster here instead.
+
+### Logs
+If your bug includes a crash, please head over to the [incredible bugreport to markdown converter](https://teamnewpipe.github.io/CrashReportToMarkdown/). Copy the result. Paste it here:
+
+<!-- That's right, here! -->

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,28 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+labels: enhancement
+assignees: ''
+
+---
+<!-- Hey. Our contribution guidelines (https://github.com/TeamNewPipe/NewPipe/blob/HEAD/.github/CONTRIBUTING.md) might be an appropriate
+ document to read before you fill out the request :) -->
+#### Is your feature request related to a problem? Please describe it
+A clear and concise description of what the problem is.
+Example: *I want to do X, but there is no way to do it.*
+
+#### Describe the solution you'd like
+A clear and concise description of what you want to happen.
+Example: *I think it would be nice if you add feature Y which makes X possible.*
+
+#### (Optional) Describe alternatives you've considered
+A clear and concise description of any alternative solutions or features you've considered.
+Example: *I considered Z, but that didn't turn out to be a good idea because...*
+
+#### Additional context
+Add any other context or screenshots about the feature request here.
+Example: *Here's a photo of my cat!*
+
+#### How will you/everyone benefit from this feature?
+Convince us! How does it change your NewPipe experience and/or your life?
+The better this paragraph is, the more likely a developer will think about working on it.

.github/PULL_REQUEST_TEMPLATE.md

@@ -1 +1,26 @@
+<!-- Hey there. Thank you so much for improving NewPipe. Please take a moment to fill out the following suggestion on how to structure this PR description. Having roughly the same layout helps everyone considerably :)-->
+
+#### What is it?
+- [ ] Bug fix
+- [ ] Feature
+
+#### Long description of the changes in your PR
+<!-- While bullet points are the norm in this section, feel free to write a text instead if you can't fit it in a list -->
+- record videos
+- create clones
+- take over the world
+
+#### Fixes the following issue(s)
+<!-- Also add reddit or other links which are relevant to your change. -->
+- 
+
+#### Relies on the following changes
+<!-- Delete this if it doesn't apply to you. -->
+- 
+
+#### Testing apk
+<!-- Ensure that you have your changes on a new branch which has a meaningful name. This name will be used as a suffix for the app ID to allow installing and testing multiple versions of NewPipe. Do NOT name your branches like "patch-0" and "feature-1". For example, if your PR implements a bug fix for comments, an appropriate branch name would be "commentfix". -->
+debug.zip
+
+#### Agreement
 - [ ] I carefully read the [contribution guidelines](https://github.com/TeamNewPipe/NewPipe/blob/HEAD/.github/CONTRIBUTING.md) and agree to them.

app/build.gradle

@@ -9,13 +9,20 @@ android {
 
     defaultConfig {
         applicationId "org.schabi.newpipe"
+        resValue "string", "app_name", "NewPipe"
         minSdkVersion 19
         targetSdkVersion 28
-        versionCode 840
-        versionName "0.18.4"
+        versionCode 910
+        versionName "0.19.1"
 
         testInstrumentationRunner "androidx.test.runner.AndroidJUnitRunner"
         vectorDrawables.useSupportLibrary = true
+
+        javaCompileOptions {
+            annotationProcessorOptions {
+                arguments = ["room.schemaLocation": "$projectDir/schemas".toString()]
+            }
+        }
     }
 
     buildTypes {
@@ -28,7 +35,18 @@ android {
         debug {
             multiDexEnabled true
             debuggable true
-            applicationIdSuffix ".debug"
+
+            // suffix the app id and the app name with git branch name
+            def workingBranch = getGitWorkingBranch()
+            def normalizedWorkingBranch = workingBranch.replaceAll("[^A-Za-z]+", "").toLowerCase()
+            if (normalizedWorkingBranch.isEmpty() || workingBranch == "master" || workingBranch == "dev") {
+                // default values when branch name could not be determined or is master or dev
+                applicationIdSuffix ".debug"
+                resValue "string", "app_name", "NewPipe Debug"
+            } else {
+                applicationIdSuffix ".debug." + normalizedWorkingBranch
+                resValue "string", "app_name", "NewPipe " + workingBranch
+            }
         }
     }
 
@@ -43,6 +61,15 @@ android {
         sourceCompatibility JavaVersion.VERSION_1_8
         targetCompatibility JavaVersion.VERSION_1_8
     }
+
+    // Required and used only by groupie
+    androidExtensions {
+        experimental = true
+    }
+
+    sourceSets {
+        androidTest.assets.srcDirs += files("$projectDir/schemas".toString())
+    }
 }
 
 ext {
@@ -59,11 +86,13 @@ ext {
 dependencies {
     implementation "org.jetbrains.kotlin:kotlin-stdlib-jdk7:$kotlin_version"
 
+    androidTestImplementation 'androidx.test.ext:junit:1.1.1'
+    androidTestImplementation "android.arch.persistence.room:testing:1.1.1"
     androidTestImplementation('androidx.test.espresso:espresso-core:3.1.0', {
         exclude module: 'support-annotations'
     })
 
-    implementation 'com.github.TeamNewPipe:NewPipeExtractor:v0.18.4'
+    implementation 'com.github.TeamNewPipe:NewPipeExtractor:69e0624e3'
     testImplementation 'junit:junit:4.12'
     testImplementation 'org.mockito:mockito-core:2.23.0'
 
@@ -75,6 +104,13 @@ dependencies {
     implementation "androidx.cardview:cardview:${androidxLibVersion}"
     implementation 'androidx.constraintlayout:constraintlayout:1.1.3'
 
+    implementation 'com.xwray:groupie:2.7.0'
+    implementation 'com.xwray:groupie-kotlin-android-extensions:2.7.0'
+
+    implementation 'androidx.lifecycle:lifecycle-livedata:2.0.0'
+    implementation 'androidx.lifecycle:lifecycle-viewmodel:2.0.0'
+    implementation 'androidx.lifecycle:lifecycle-extensions:2.0.0'
+
     // Originally in NewPipeExtractor
     implementation 'com.grack:nanojson:1.1'
     implementation 'org.jsoup:jsoup:1.9.2'
@@ -113,3 +149,19 @@ dependencies {
     implementation "io.noties.markwon:core:${markwonVersion}"
     implementation "io.noties.markwon:linkify:${markwonVersion}"
 }
+
+static String getGitWorkingBranch() {
+    try {
+        def gitProcess = "git rev-parse --abbrev-ref HEAD".execute()
+        gitProcess.waitFor()
+        if (gitProcess.exitValue() == 0) {
+            return gitProcess.text.trim()
+        } else {
+            // not a git repository
+            return ""
+        }
+    } catch (IOException ignored) {
+        // git was not found
+        return ""
+    }
+}

app/src/androidTest/java/org/schabi/newpipe/database/AppDatabaseTest.kt

@@ -0,0 +1,118 @@
+package org.schabi.newpipe.database
+
+import android.content.ContentValues
+import android.database.sqlite.SQLiteDatabase
+import androidx.room.Room
+import androidx.room.testing.MigrationTestHelper
+import androidx.sqlite.db.framework.FrameworkSQLiteOpenHelperFactory
+import androidx.test.core.app.ApplicationProvider
+import androidx.test.ext.junit.runners.AndroidJUnit4
+import androidx.test.platform.app.InstrumentationRegistry
+import org.junit.Assert.assertEquals
+import org.junit.Assert.assertNull
+import org.junit.Rule
+import org.junit.Test
+import org.junit.runner.RunWith
+import org.schabi.newpipe.extractor.stream.StreamType
+
+@RunWith(AndroidJUnit4::class)
+class AppDatabaseTest {
+    companion object {
+        private const val DEFAULT_SERVICE_ID = 0
+        private const val DEFAULT_URL = "https://www.youtube.com/watch?v=cDphUib5iG4"
+        private const val DEFAULT_TITLE = "Test Title"
+        private val DEFAULT_TYPE = StreamType.VIDEO_STREAM
+        private const val DEFAULT_DURATION = 480L
+        private const val DEFAULT_UPLOADER_NAME = "Uploader Test"
+        private const val DEFAULT_THUMBNAIL = "https://example.com/example.jpg"
+
+        private const val DEFAULT_SECOND_SERVICE_ID = 0
+        private const val DEFAULT_SECOND_URL = "https://www.youtube.com/watch?v=ncQU6iBn5Fc"
+    }
+
+    @get:Rule val testHelper = MigrationTestHelper(InstrumentationRegistry.getInstrumentation(),
+            AppDatabase::class.java.canonicalName, FrameworkSQLiteOpenHelperFactory());
+
+    @Test
+    fun migrateDatabaseFrom2to3() {
+        val databaseInV2 = testHelper.createDatabase(AppDatabase.DATABASE_NAME, Migrations.DB_VER_2)
+
+        databaseInV2.run {
+            insert("streams", SQLiteDatabase.CONFLICT_FAIL, ContentValues().apply {
+                // put("uid", null)
+                put("service_id", DEFAULT_SERVICE_ID)
+                put("url", DEFAULT_URL)
+                put("title", DEFAULT_TITLE)
+                put("stream_type", DEFAULT_TYPE.name)
+                put("duration", DEFAULT_DURATION)
+                put("uploader", DEFAULT_UPLOADER_NAME)
+                put("thumbnail_url", DEFAULT_THUMBNAIL)
+            })
+            insert("streams", SQLiteDatabase.CONFLICT_FAIL, ContentValues().apply {
+                // put("uid", null)
+                put("service_id", DEFAULT_SECOND_SERVICE_ID)
+                put("url", DEFAULT_SECOND_URL)
+                // put("title", null)
+                // put("stream_type", null)
+                // put("duration", null)
+                // put("uploader", null)
+                // put("thumbnail_url", null)
+            })
+            insert("streams", SQLiteDatabase.CONFLICT_FAIL, ContentValues().apply {
+                // put("uid", null)
+                put("service_id", DEFAULT_SERVICE_ID)
+                // put("url", null)
+                // put("title", null)
+                // put("stream_type", null)
+                // put("duration", null)
+                // put("uploader", null)
+                // put("thumbnail_url", null)
+            })
+            close()
+        }
+
+        testHelper.runMigrationsAndValidate(AppDatabase.DATABASE_NAME, Migrations.DB_VER_3,
+                true, Migrations.MIGRATION_2_3);
+
+        val migratedDatabaseV3 = getMigratedDatabase()
+        val listFromDB = migratedDatabaseV3.streamDAO().all.blockingFirst()
+
+        // Only expect 2, the one with the null url will be ignored
+        assertEquals(2, listFromDB.size)
+
+        val streamFromMigratedDatabase = listFromDB[0]
+        assertEquals(DEFAULT_SERVICE_ID, streamFromMigratedDatabase.serviceId)
+        assertEquals(DEFAULT_URL, streamFromMigratedDatabase.url)
+        assertEquals(DEFAULT_TITLE, streamFromMigratedDatabase.title)
+        assertEquals(DEFAULT_TYPE, streamFromMigratedDatabase.streamType)
+        assertEquals(DEFAULT_DURATION, streamFromMigratedDatabase.duration)
+        assertEquals(DEFAULT_UPLOADER_NAME, streamFromMigratedDatabase.uploader)
+        assertEquals(DEFAULT_THUMBNAIL, streamFromMigratedDatabase.thumbnailUrl)
+        assertNull(streamFromMigratedDatabase.viewCount)
+        assertNull(streamFromMigratedDatabase.textualUploadDate)
+        assertNull(streamFromMigratedDatabase.uploadDate)
+        assertNull(streamFromMigratedDatabase.isUploadDateApproximation)
+
+        val secondStreamFromMigratedDatabase = listFromDB[1]
+        assertEquals(DEFAULT_SECOND_SERVICE_ID, secondStreamFromMigratedDatabase.serviceId)
+        assertEquals(DEFAULT_SECOND_URL, secondStreamFromMigratedDatabase.url)
+        assertEquals("", secondStreamFromMigratedDatabase.title)
+        // Should fallback to VIDEO_STREAM
+        assertEquals(StreamType.VIDEO_STREAM, secondStreamFromMigratedDatabase.streamType)
+        assertEquals(0, secondStreamFromMigratedDatabase.duration)
+        assertEquals("", secondStreamFromMigratedDatabase.uploader)
+        assertEquals("", secondStreamFromMigratedDatabase.thumbnailUrl)
+        assertNull(secondStreamFromMigratedDatabase.viewCount)
+        assertNull(secondStreamFromMigratedDatabase.textualUploadDate)
+        assertNull(secondStreamFromMigratedDatabase.uploadDate)
+        assertNull(secondStreamFromMigratedDatabase.isUploadDateApproximation)
+    }
+
+    private fun getMigratedDatabase(): AppDatabase {
+        val database: AppDatabase = Room.databaseBuilder(ApplicationProvider.getApplicationContext(),
+                AppDatabase::class.java, AppDatabase.DATABASE_NAME)
+                .build()
+        testHelper.closeWhenFinished(database)
+        return database
+    }
+}

app/src/debug/AndroidManifest.xml

@@ -6,12 +6,5 @@
 
     <application
         android:name=".DebugApp"
-        android:label="NewPipe Debug"
-        tools:replace="android:name, android:label">
-        <activity
-            android:name=".MainActivity"
-            android:label="NewPipe Debug"
-            tools:replace="android:label"/>
-    </application>
-
+        tools:replace="android:name" />
 </manifest>

app/src/main/AndroidManifest.xml

@@ -61,7 +61,8 @@
             android:name=".player.MainVideoPlayer"
             android:configChanges="keyboard|keyboardHidden|orientation|screenSize"
             android:label="@string/app_name"
-            android:launchMode="singleTask"/>
+            android:launchMode="singleTask"
+            android:theme="@style/VideoPlayerTheme"/>
 
         <activity
             android:name=".settings.SettingsActivity"
@@ -73,6 +74,7 @@
 
         <service android:name=".local.subscription.services.SubscriptionsImportService"/>
         <service android:name=".local.subscription.services.SubscriptionsExportService"/>
+        <service android:name=".local.feed.service.FeedLoadService"/>
 
         <activity
             android:name=".PanicResponderActivity"
@@ -145,6 +147,7 @@
                 <data android:host="youtube.com"/>
                 <data android:host="m.youtube.com"/>
                 <data android:host="www.youtube.com"/>
+                <data android:host="music.youtube.com"/>
                 <!-- video prefix -->
                 <data android:pathPrefix="/v/"/>
                 <data android:pathPrefix="/embed/"/>
@@ -243,14 +246,7 @@
                 <data android:host="tube.poal.co"/>
                 <data android:host="invidious.13ad.de"/>
                 <data android:host="yt.elukerio.org"/>
-                <!-- video prefix -->
-                <data android:pathPrefix="/embed/"/>
-                <data android:pathPrefix="/watch"/>
-                <!-- channel prefix -->
-                <data android:pathPrefix="/channel/"/>
-                <data android:pathPrefix="/user/"/>
-                <!-- playlist prefix -->
-                <data android:pathPrefix="/playlist"/>
+                <data android:pathPrefix="/"/>
             </intent-filter>
 
             <!-- Soundcloud filter -->
@@ -276,8 +272,26 @@
                 <category android:name="android.intent.category.DEFAULT"/>
                 <data android:mimeType="text/plain"/>
             </intent-filter>
-        </activity>
 
+            <!-- MediaCCC filter -->
+            <intent-filter>
+                <action android:name="android.intent.action.VIEW"/>
+                <action android:name="android.media.action.MEDIA_PLAY_FROM_SEARCH"/>
+                <action android:name="android.nfc.action.NDEF_DISCOVERED"/>
+
+                <category android:name="android.intent.category.DEFAULT"/>
+                <category android:name="android.intent.category.BROWSABLE"/>
+
+                <data android:scheme="http"/>
+                <data android:scheme="https"/>
+                <data android:host="media.ccc.de"/>
+                <!-- video prefix -->
+                <data android:pathPrefix="/v/"/>
+                <!-- channel prefix-->
+                <data android:pathPrefix="/c/"/>
+                <data android:pathPrefix="/b/"/>
+            </intent-filter>
+        </activity>
         <service
             android:name=".RouterActivity$FetcherService"
             android:exported="false"/>

app/src/main/java/org/schabi/newpipe/DownloaderImpl.java

@@ -38,7 +38,7 @@ import okhttp3.ResponseBody;
 import static org.schabi.newpipe.MainActivity.DEBUG;
 
 public class DownloaderImpl extends Downloader {
-    public static final String USER_AGENT = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0";
+    public static final String USER_AGENT = "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Firefox/68.0";
 
     private static DownloaderImpl instance;
     private String mCookies;
@@ -171,7 +171,8 @@ public class DownloaderImpl extends Downloader {
             responseBodyToReturn = body.string();
         }
 
-        return new Response(response.code(), response.message(), response.headers().toMultimap(), responseBodyToReturn);
+        final String latestUrl = response.request().url().toString();
+        return new Response(response.code(), response.message(), response.headers().toMultimap(), responseBodyToReturn, latestUrl);
     }
 
     /**

app/src/main/java/org/schabi/newpipe/MainActivity.java

@@ -159,7 +159,7 @@ public class MainActivity extends AppCompatActivity {
                 .add(R.id.menu_tabs_group, ITEM_ID_SUBSCRIPTIONS, ORDER, R.string.tab_subscriptions)
                 .setIcon(ThemeHelper.resolveResourceIdFromAttr(this, R.attr.ic_channel));
         drawerItems.getMenu()
-                .add(R.id.menu_tabs_group, ITEM_ID_FEED, ORDER, R.string.fragment_whats_new)
+                .add(R.id.menu_tabs_group, ITEM_ID_FEED, ORDER, R.string.fragment_feed_title)
                 .setIcon(ThemeHelper.resolveResourceIdFromAttr(this, R.attr.rss));
         drawerItems.getMenu()
                 .add(R.id.menu_tabs_group, ITEM_ID_BOOKMARKS, ORDER, R.string.tab_bookmarks)
@@ -240,7 +240,7 @@ public class MainActivity extends AppCompatActivity {
                 NavigationHelper.openSubscriptionFragment(getSupportFragmentManager());
                 break;
             case ITEM_ID_FEED:
-                NavigationHelper.openWhatsNewFragment(getSupportFragmentManager());
+                NavigationHelper.openFeedFragment(getSupportFragmentManager());
                 break;
             case ITEM_ID_BOOKMARKS:
                 NavigationHelper.openBookmarksFragment(getSupportFragmentManager());
@@ -389,7 +389,7 @@ public class MainActivity extends AppCompatActivity {
                 .add(R.id.menu_tabs_group, ITEM_ID_SUBSCRIPTIONS, ORDER, R.string.tab_subscriptions)
                 .setIcon(ThemeHelper.resolveResourceIdFromAttr(this, R.attr.ic_channel));
         drawerItems.getMenu()
-                .add(R.id.menu_tabs_group, ITEM_ID_FEED, ORDER, R.string.fragment_whats_new)
+                .add(R.id.menu_tabs_group, ITEM_ID_FEED, ORDER, R.string.fragment_feed_title)
                 .setIcon(ThemeHelper.resolveResourceIdFromAttr(this, R.attr.rss));
         drawerItems.getMenu()
                 .add(R.id.menu_tabs_group, ITEM_ID_BOOKMARKS, ORDER, R.string.tab_bookmarks)

app/src/main/java/org/schabi/newpipe/NewPipeDatabase.java

@@ -1,13 +1,16 @@
 package org.schabi.newpipe;
 
-import androidx.room.Room;
 import android.content.Context;
+import android.database.Cursor;
+
 import androidx.annotation.NonNull;
+import androidx.room.Room;
 
 import org.schabi.newpipe.database.AppDatabase;
 
 import static org.schabi.newpipe.database.AppDatabase.DATABASE_NAME;
-import static org.schabi.newpipe.database.Migrations.MIGRATION_11_12;
+import static org.schabi.newpipe.database.Migrations.MIGRATION_1_2;
+import static org.schabi.newpipe.database.Migrations.MIGRATION_2_3;
 
 public final class NewPipeDatabase {
 
@@ -20,8 +23,7 @@ public final class NewPipeDatabase {
     private static AppDatabase getDatabase(Context context) {
         return Room
                 .databaseBuilder(context.getApplicationContext(), AppDatabase.class, DATABASE_NAME)
-                .addMigrations(MIGRATION_11_12)
-                .fallbackToDestructiveMigration()
+                .addMigrations(MIGRATION_1_2, MIGRATION_2_3)
                 .build();
     }
 
@@ -39,4 +41,14 @@ public final class NewPipeDatabase {
 
         return result;
     }
+
+    public static void checkpoint() {
+        if (databaseInstance == null) {
+            throw new IllegalStateException("database is not initialized");
+        }
+        Cursor c = databaseInstance.query("pragma wal_checkpoint(full)", null);
+        if (c.moveToFirst() && c.getInt(0) == 1) {
+            throw new RuntimeException("Checkpoint was blocked from completing");
+        }
+    }
 }

app/src/main/java/org/schabi/newpipe/RouterActivity.java

@@ -9,12 +9,6 @@ import android.content.SharedPreferences;
 import android.content.pm.PackageManager;
 import android.os.Bundle;
 import android.preference.PreferenceManager;
-import androidx.annotation.DrawableRes;
-import androidx.annotation.NonNull;
-import androidx.annotation.Nullable;
-import androidx.core.app.NotificationCompat;
-import androidx.appcompat.app.AlertDialog;
-import androidx.appcompat.app.AppCompatActivity;
 import android.text.TextUtils;
 import android.view.ContextThemeWrapper;
 import android.view.LayoutInflater;
@@ -26,6 +20,12 @@ import android.widget.RadioButton;
 import android.widget.RadioGroup;
 import android.widget.Toast;
 
+import androidx.annotation.DrawableRes;
+import androidx.annotation.NonNull;
+import androidx.annotation.Nullable;
+import androidx.appcompat.app.AlertDialog;
+import androidx.appcompat.app.AppCompatActivity;
+import androidx.core.app.NotificationCompat;
 import androidx.fragment.app.FragmentManager;
 
 import org.schabi.newpipe.download.DownloadDialog;
@@ -49,12 +49,11 @@ import org.schabi.newpipe.util.ListHelper;
 import org.schabi.newpipe.util.NavigationHelper;
 import org.schabi.newpipe.util.PermissionHelper;
 import org.schabi.newpipe.util.ThemeHelper;
+import org.schabi.newpipe.util.urlfinder.UrlFinder;
 
 import java.io.Serializable;
 import java.util.ArrayList;
 import java.util.Arrays;
-import java.util.Collection;
-import java.util.HashSet;
 import java.util.List;
 
 import icepick.Icepick;
@@ -625,78 +624,18 @@ public class RouterActivity extends AppCompatActivity {
     // Utils
     //////////////////////////////////////////////////////////////////////////*/
 
-    /**
-     * Removes invisible separators (\p{Z}) and punctuation characters including
-     * brackets (\p{P}). See http://www.regular-expressions.info/unicode.html for
-     * more details.
-     */
-    private final static String REGEX_REMOVE_FROM_URL = "[\\p{Z}\\p{P}]";
-
+    @Nullable
     private String getUrl(Intent intent) {
-        // first gather data and find service
-        String videoUrl = null;
+        String foundUrl = null;
         if (intent.getData() != null) {
-            // this means the video was called though another app
-            videoUrl = intent.getData().toString();
+            // Called from another app
+            foundUrl = intent.getData().toString();
         } else if (intent.getStringExtra(Intent.EXTRA_TEXT) != null) {
-            //this means that vidoe was called through share menu
-            String extraText = intent.getStringExtra(Intent.EXTRA_TEXT);
-            final String[] uris = getUris(extraText);
-            videoUrl = uris.length > 0 ? uris[0] : null;
+            // Called from the share menu
+            final String extraText = intent.getStringExtra(Intent.EXTRA_TEXT);
+            foundUrl = UrlFinder.firstUrlFromInput(extraText);
         }
 
-        return videoUrl;
-    }
-
-    private String removeHeadingGibberish(final String input) {
-        int start = 0;
-        for (int i = input.indexOf("://") - 1; i >= 0; i--) {
-            if (!input.substring(i, i + 1).matches("\\p{L}")) {
-                start = i + 1;
-                break;
-            }
-        }
-        return input.substring(start, input.length());
-    }
-
-    private String trim(final String input) {
-        if (input == null || input.length() < 1) {
-            return input;
-        } else {
-            String output = input;
-            while (output.length() > 0 && output.substring(0, 1).matches(REGEX_REMOVE_FROM_URL)) {
-                output = output.substring(1);
-            }
-            while (output.length() > 0
-                    && output.substring(output.length() - 1, output.length()).matches(REGEX_REMOVE_FROM_URL)) {
-                output = output.substring(0, output.length() - 1);
-            }
-            return output;
-        }
-    }
-
-    /**
-     * Retrieves all Strings which look remotely like URLs from a text.
-     * Used if NewPipe was called through share menu.
-     *
-     * @param sharedText text to scan for URLs.
-     * @return potential URLs
-     */
-    protected String[] getUris(final String sharedText) {
-        final Collection<String> result = new HashSet<>();
-        if (sharedText != null) {
-            final String[] array = sharedText.split("\\p{Space}");
-            for (String s : array) {
-                s = trim(s);
-                if (s.length() != 0) {
-                    if (s.matches(".+://.+")) {
-                        result.add(removeHeadingGibberish(s));
-                    } else if (s.matches(".+\\..+")) {
-                        result.add("http://" + s);
-                    }
-                }
-            }
-        }
-        return result.toArray(new String[result.size()]);
+        return foundUrl;
     }
 }

app/src/main/java/org/schabi/newpipe/about/AboutActivity.java

@@ -45,7 +45,8 @@ public class AboutActivity extends AppCompatActivity {
             new SoftwareComponent("RxJava", "2016 - 2020", "RxJava Contributors", "https://github.com/ReactiveX/RxJava", StandardLicenses.APACHE2),
             new SoftwareComponent("RxBinding", "2015 - 2018", "Jake Wharton", "https://github.com/JakeWharton/RxBinding", StandardLicenses.APACHE2),
             new SoftwareComponent("PrettyTime", "2012 - 2020", "Lincoln Baxter, III", "https://github.com/ocpsoft/prettytime", StandardLicenses.APACHE2),
-            new SoftwareComponent("Markwon", "2017 - 2020", "Noties", "https://github.com/noties/Markwon", StandardLicenses.APACHE2)
+            new SoftwareComponent("Markwon", "2017 - 2020", "Noties", "https://github.com/noties/Markwon", StandardLicenses.APACHE2),
+            new SoftwareComponent("Groupie", "2016", "Lisa Wray", "https://github.com/lisawray/groupie", StandardLicenses.MIT)
     };
 
     /**

app/src/main/java/org/schabi/newpipe/database/AppDatabase.java

@@ -4,6 +4,12 @@ import androidx.room.Database;
 import androidx.room.RoomDatabase;
 import androidx.room.TypeConverters;
 
+import org.schabi.newpipe.database.feed.dao.FeedDAO;
+import org.schabi.newpipe.database.feed.dao.FeedGroupDAO;
+import org.schabi.newpipe.database.feed.model.FeedEntity;
+import org.schabi.newpipe.database.feed.model.FeedGroupEntity;
+import org.schabi.newpipe.database.feed.model.FeedGroupSubscriptionEntity;
+import org.schabi.newpipe.database.feed.model.FeedLastUpdatedEntity;
 import org.schabi.newpipe.database.history.dao.SearchHistoryDAO;
 import org.schabi.newpipe.database.history.dao.StreamHistoryDAO;
 import org.schabi.newpipe.database.history.model.SearchHistoryEntry;
@@ -21,35 +27,33 @@ import org.schabi.newpipe.database.stream.model.StreamStateEntity;
 import org.schabi.newpipe.database.subscription.SubscriptionDAO;
 import org.schabi.newpipe.database.subscription.SubscriptionEntity;
 
-import static org.schabi.newpipe.database.Migrations.DB_VER_12_0;
+import static org.schabi.newpipe.database.Migrations.DB_VER_3;
 
 @TypeConverters({Converters.class})
 @Database(
         entities = {
                 SubscriptionEntity.class, SearchHistoryEntry.class,
                 StreamEntity.class, StreamHistoryEntity.class, StreamStateEntity.class,
-                PlaylistEntity.class, PlaylistStreamEntity.class, PlaylistRemoteEntity.class
+                PlaylistEntity.class, PlaylistStreamEntity.class, PlaylistRemoteEntity.class,
+                FeedEntity.class, FeedGroupEntity.class, FeedGroupSubscriptionEntity.class,
+                FeedLastUpdatedEntity.class
         },
-        version = DB_VER_12_0,
-        exportSchema = false
+        version = DB_VER_3
 )
 public abstract class AppDatabase extends RoomDatabase {
-
     public static final String DATABASE_NAME = "newpipe.db";
 
-    public abstract SubscriptionDAO subscriptionDAO();
-
     public abstract SearchHistoryDAO searchHistoryDAO();
 
     public abstract StreamDAO streamDAO();
-
     public abstract StreamHistoryDAO streamHistoryDAO();
-
     public abstract StreamStateDAO streamStateDAO();
 
     public abstract PlaylistDAO playlistDAO();
-
     public abstract PlaylistStreamDAO playlistStreamDAO();
-
     public abstract PlaylistRemoteDAO playlistRemoteDAO();
+
+    public abstract FeedDAO feedDAO();
+    public abstract FeedGroupDAO feedGroupDAO();
+    public abstract SubscriptionDAO subscriptionDAO();
 }

app/src/main/java/org/schabi/newpipe/database/Converters.java

@@ -3,6 +3,7 @@ package org.schabi.newpipe.database;
 import androidx.room.TypeConverter;
 
 import org.schabi.newpipe.extractor.stream.StreamType;
+import org.schabi.newpipe.local.subscription.FeedGroupIcon;
 
 import java.util.Date;
 
@@ -37,4 +38,18 @@ public class Converters {
     public static String stringOf(StreamType streamType) {
         return streamType.name();
     }
+
+    @TypeConverter
+    public static Integer integerOf(FeedGroupIcon feedGroupIcon) {
+        return feedGroupIcon.getId();
+    }
+
+    @TypeConverter
+    public static FeedGroupIcon feedGroupIconOf(Integer id) {
+        for (FeedGroupIcon icon : FeedGroupIcon.values()) {
+            if (icon.getId() == id) return icon;
+        }
+
+        throw new IllegalArgumentException("There's no feed group icon with the id \"" + id + "\"");
+    }
 }

app/src/main/java/org/schabi/newpipe/database/Migrations.java

@@ -8,14 +8,14 @@ import android.util.Log;
 import org.schabi.newpipe.BuildConfig;
 
 public class Migrations {
-
-    public static final int DB_VER_11_0 = 1;
-    public static final int DB_VER_12_0 = 2;
+    public static final int DB_VER_1 = 1;
+    public static final int DB_VER_2 = 2;
+    public static final int DB_VER_3 = 3;
 
     public static final boolean DEBUG = !BuildConfig.BUILD_TYPE.equals("release");
     private static final String TAG = Migrations.class.getName();
 
-    public static final Migration MIGRATION_11_12 = new Migration(DB_VER_11_0, DB_VER_12_0) {
+    public static final Migration MIGRATION_1_2 = new Migration(DB_VER_1, DB_VER_2) {
         @Override
         public void migrate(@NonNull SupportSQLiteDatabase database) {
             if(DEBUG) {
@@ -71,4 +71,40 @@ public class Migrations {
             }
         }
     };
+
+    public static final Migration MIGRATION_2_3 = new Migration(DB_VER_2, DB_VER_3) {
+        @Override
+        public void migrate(@NonNull SupportSQLiteDatabase database) {
+            // Add NOT NULLs and new fields
+            database.execSQL("CREATE TABLE IF NOT EXISTS streams_new " +
+                    "(uid INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL, service_id INTEGER NOT NULL, url TEXT NOT NULL, title TEXT NOT NULL, stream_type TEXT NOT NULL," +
+                    " duration INTEGER NOT NULL, uploader TEXT NOT NULL, thumbnail_url TEXT, view_count INTEGER, textual_upload_date TEXT, upload_date INTEGER," +
+                    " is_upload_date_approximation INTEGER)");
+
+            database.execSQL("INSERT INTO streams_new (uid, service_id, url, title, stream_type," +
+                    "duration, uploader, thumbnail_url, view_count," +
+                    "textual_upload_date, upload_date, is_upload_date_approximation) " +
+
+                    "SELECT uid, service_id, url, ifnull(title, ''), ifnull(stream_type, 'VIDEO_STREAM')," +
+                    "ifnull(duration, 0), ifnull(uploader, ''), ifnull(thumbnail_url, ''), NULL," +
+                    "NULL, NULL, NULL " +
+
+                    "FROM streams " +
+                    "WHERE url IS NOT NULL");
+
+            database.execSQL("DROP TABLE streams");
+            database.execSQL("ALTER TABLE streams_new RENAME TO streams");
+            database.execSQL("CREATE UNIQUE INDEX index_streams_service_id_url ON streams (service_id, url)");
+
+            // Tables for feed feature
+            database.execSQL("CREATE TABLE IF NOT EXISTS feed (stream_id INTEGER NOT NULL, subscription_id INTEGER NOT NULL, PRIMARY KEY(stream_id, subscription_id), FOREIGN KEY(stream_id) REFERENCES streams(uid) ON UPDATE CASCADE ON DELETE CASCADE DEFERRABLE INITIALLY DEFERRED, FOREIGN KEY(subscription_id) REFERENCES subscriptions(uid) ON UPDATE CASCADE ON DELETE CASCADE DEFERRABLE INITIALLY DEFERRED)");
+            database.execSQL("CREATE INDEX index_feed_subscription_id ON feed (subscription_id)");
+            database.execSQL("CREATE TABLE IF NOT EXISTS feed_group (uid INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL, name TEXT NOT NULL, icon_id INTEGER NOT NULL, sort_order INTEGER NOT NULL)");
+            database.execSQL("CREATE INDEX index_feed_group_sort_order ON feed_group (sort_order)");
+            database.execSQL("CREATE TABLE IF NOT EXISTS feed_group_subscription_join (group_id INTEGER NOT NULL, subscription_id INTEGER NOT NULL, PRIMARY KEY(group_id, subscription_id), FOREIGN KEY(group_id) REFERENCES feed_group(uid) ON UPDATE CASCADE ON DELETE CASCADE DEFERRABLE INITIALLY DEFERRED, FOREIGN KEY(subscription_id) REFERENCES subscriptions(uid) ON UPDATE CASCADE ON DELETE CASCADE DEFERRABLE INITIALLY DEFERRED)");
+            database.execSQL("CREATE INDEX index_feed_group_subscription_join_subscription_id ON feed_group_subscription_join (subscription_id)");
+            database.execSQL("CREATE TABLE IF NOT EXISTS feed_last_updated (subscription_id INTEGER NOT NULL, last_updated INTEGER, PRIMARY KEY(subscription_id), FOREIGN KEY(subscription_id) REFERENCES subscriptions(uid) ON UPDATE CASCADE ON DELETE CASCADE DEFERRABLE INITIALLY DEFERRED)");
+        }
+    };
+
 }

app/src/main/java/org/schabi/newpipe/database/feed/dao/FeedDAO.kt

@@ -0,0 +1,147 @@
+package org.schabi.newpipe.database.feed.dao
+
+import androidx.room.*
+import io.reactivex.Flowable
+import org.schabi.newpipe.database.feed.model.FeedEntity
+import org.schabi.newpipe.database.feed.model.FeedLastUpdatedEntity
+import org.schabi.newpipe.database.stream.model.StreamEntity
+import org.schabi.newpipe.database.subscription.SubscriptionEntity
+import java.util.*
+
+@Dao
+abstract class FeedDAO {
+    @Query("DELETE FROM feed")
+    abstract fun deleteAll(): Int
+
+    @Query("""
+        SELECT s.* FROM streams s
+
+        INNER JOIN feed f
+        ON s.uid = f.stream_id
+
+        ORDER BY s.upload_date IS NULL DESC, s.upload_date DESC, s.uploader ASC
+
+        LIMIT 500
+        """)
+    abstract fun getAllStreams(): Flowable<List<StreamEntity>>
+
+    @Query("""
+        SELECT s.* FROM streams s
+
+        INNER JOIN feed f
+        ON s.uid = f.stream_id
+
+        INNER JOIN feed_group_subscription_join fgs
+        ON fgs.subscription_id = f.subscription_id
+
+        INNER JOIN feed_group fg
+        ON fg.uid = fgs.group_id
+
+        WHERE fgs.group_id = :groupId
+
+        ORDER BY s.upload_date IS NULL DESC, s.upload_date DESC, s.uploader ASC
+        LIMIT 500
+        """)
+    abstract fun getAllStreamsFromGroup(groupId: Long): Flowable<List<StreamEntity>>
+
+    @Query("""
+        DELETE FROM feed WHERE
+
+        feed.stream_id IN (
+            SELECT s.uid FROM streams s
+
+            INNER JOIN feed f
+            ON s.uid = f.stream_id
+
+            WHERE s.upload_date < :date
+        )
+        """)
+    abstract fun unlinkStreamsOlderThan(date: Date)
+
+    @Query("""
+        DELETE FROM feed
+        
+        WHERE feed.subscription_id = :subscriptionId
+
+        AND feed.stream_id IN (
+            SELECT s.uid FROM streams s
+
+            INNER JOIN feed f
+            ON s.uid = f.stream_id
+
+            WHERE s.stream_type = "LIVE_STREAM" OR s.stream_type = "AUDIO_LIVE_STREAM"
+        )
+        """)
+    abstract fun unlinkOldLivestreams(subscriptionId: Long)
+
+    @Insert(onConflict = OnConflictStrategy.IGNORE)
+    abstract fun insert(feedEntity: FeedEntity)
+
+    @Insert(onConflict = OnConflictStrategy.IGNORE)
+    abstract fun insertAll(entities: List<FeedEntity>): List<Long>
+
+    @Insert(onConflict = OnConflictStrategy.IGNORE)
+    internal abstract fun insertLastUpdated(lastUpdatedEntity: FeedLastUpdatedEntity): Long
+
+    @Update(onConflict = OnConflictStrategy.IGNORE)
+    internal abstract fun updateLastUpdated(lastUpdatedEntity: FeedLastUpdatedEntity)
+
+    @Transaction
+    open fun setLastUpdatedForSubscription(lastUpdatedEntity: FeedLastUpdatedEntity) {
+        val id = insertLastUpdated(lastUpdatedEntity)
+
+        if (id == -1L) {
+            updateLastUpdated(lastUpdatedEntity)
+        }
+    }
+
+    @Query("""
+        SELECT MIN(lu.last_updated) FROM feed_last_updated lu
+
+        INNER JOIN feed_group_subscription_join fgs
+        ON fgs.subscription_id = lu.subscription_id AND fgs.group_id = :groupId
+        """)
+    abstract fun oldestSubscriptionUpdate(groupId: Long): Flowable<List<Date>>
+
+    @Query("SELECT MIN(last_updated) FROM feed_last_updated")
+    abstract fun oldestSubscriptionUpdateFromAll(): Flowable<List<Date>>
+
+    @Query("SELECT COUNT(*) FROM feed_last_updated WHERE last_updated IS NULL")
+    abstract fun notLoadedCount(): Flowable<Long>
+
+    @Query("""
+        SELECT COUNT(*) FROM subscriptions s
+        
+        INNER JOIN feed_group_subscription_join fgs
+        ON s.uid = fgs.subscription_id AND fgs.group_id = :groupId
+
+        LEFT JOIN feed_last_updated lu
+        ON s.uid = lu.subscription_id 
+
+        WHERE lu.last_updated IS NULL
+        """)
+    abstract fun notLoadedCountForGroup(groupId: Long): Flowable<Long>
+
+    @Query("""
+        SELECT s.* FROM subscriptions s
+
+        LEFT JOIN feed_last_updated lu
+        ON s.uid = lu.subscription_id 
+
+        WHERE lu.last_updated IS NULL OR lu.last_updated < :outdatedThreshold
+        """)
+    abstract fun getAllOutdated(outdatedThreshold: Date): Flowable<List<SubscriptionEntity>>
+
+    @Query("""
+        SELECT s.* FROM subscriptions s
+
+        INNER JOIN feed_group_subscription_join fgs
+        ON s.uid = fgs.subscription_id AND fgs.group_id = :groupId
+
+        LEFT JOIN feed_last_updated lu
+        ON s.uid = lu.subscription_id
+
+        WHERE lu.last_updated IS NULL OR lu.last_updated < :outdatedThreshold
+        """)
+    abstract fun getAllOutdatedForGroup(groupId: Long, outdatedThreshold: Date): Flowable<List<SubscriptionEntity>>
+}

app/src/main/java/org/schabi/newpipe/database/feed/dao/FeedGroupDAO.kt

@@ -0,0 +1,62 @@
+package org.schabi.newpipe.database.feed.dao
+
+import androidx.room.*
+import io.reactivex.Flowable
+import io.reactivex.Maybe
+import org.schabi.newpipe.database.feed.model.FeedGroupEntity
+import org.schabi.newpipe.database.feed.model.FeedGroupSubscriptionEntity
+
+@Dao
+abstract class FeedGroupDAO {
+
+    @Query("SELECT * FROM feed_group ORDER BY sort_order ASC")
+    abstract fun getAll(): Flowable<List<FeedGroupEntity>>
+
+    @Query("SELECT * FROM feed_group WHERE uid = :groupId")
+    abstract fun getGroup(groupId: Long): Maybe<FeedGroupEntity>
+
+    @Transaction
+    open fun insert(feedGroupEntity: FeedGroupEntity): Long {
+        val nextSortOrder = nextSortOrder()
+        feedGroupEntity.sortOrder = nextSortOrder
+        return insertInternal(feedGroupEntity)
+    }
+
+    @Update(onConflict = OnConflictStrategy.IGNORE)
+    abstract fun update(feedGroupEntity: FeedGroupEntity): Int
+
+    @Query("DELETE FROM feed_group")
+    abstract fun deleteAll(): Int
+
+    @Query("DELETE FROM feed_group WHERE uid = :groupId")
+    abstract fun delete(groupId: Long): Int
+
+    @Query("SELECT subscription_id FROM feed_group_subscription_join WHERE group_id = :groupId")
+    abstract fun getSubscriptionIdsFor(groupId: Long): Flowable<List<Long>>
+
+    @Query("DELETE FROM feed_group_subscription_join WHERE group_id = :groupId")
+    abstract fun deleteSubscriptionsFromGroup(groupId: Long): Int
+
+    @Insert(onConflict = OnConflictStrategy.IGNORE)
+    abstract fun insertSubscriptionsToGroup(entities: List<FeedGroupSubscriptionEntity>): List<Long>
+
+    @Transaction
+    open fun updateSubscriptionsForGroup(groupId: Long, subscriptionIds: List<Long>) {
+        deleteSubscriptionsFromGroup(groupId)
+        insertSubscriptionsToGroup(subscriptionIds.map { FeedGroupSubscriptionEntity(groupId, it) })
+    }
+
+    @Transaction
+    open fun updateOrder(orderMap: Map<Long, Long>) {
+        orderMap.forEach { (groupId, sortOrder) -> updateOrder(groupId, sortOrder) }
+    }
+
+    @Query("UPDATE feed_group SET sort_order = :sortOrder WHERE uid = :groupId")
+    abstract fun updateOrder(groupId: Long, sortOrder: Long): Int
+
+    @Query("SELECT IFNULL(MAX(sort_order) + 1, 0) FROM feed_group")
+    protected abstract fun nextSortOrder(): Long
+
+    @Insert(onConflict = OnConflictStrategy.ABORT)
+    protected abstract fun insertInternal(feedGroupEntity: FeedGroupEntity): Long
+}

app/src/main/java/org/schabi/newpipe/database/feed/model/FeedEntity.kt

@@ -0,0 +1,43 @@
+package org.schabi.newpipe.database.feed.model
+
+import androidx.room.ColumnInfo
+import androidx.room.Entity
+import androidx.room.ForeignKey
+import androidx.room.Index
+import org.schabi.newpipe.database.feed.model.FeedEntity.Companion.FEED_TABLE
+import org.schabi.newpipe.database.feed.model.FeedEntity.Companion.STREAM_ID
+import org.schabi.newpipe.database.feed.model.FeedEntity.Companion.SUBSCRIPTION_ID
+import org.schabi.newpipe.database.stream.model.StreamEntity
+import org.schabi.newpipe.database.subscription.SubscriptionEntity
+
+@Entity(tableName = FEED_TABLE,
+        primaryKeys = [STREAM_ID, SUBSCRIPTION_ID],
+        indices = [Index(SUBSCRIPTION_ID)],
+        foreignKeys = [
+            ForeignKey(
+                    entity = StreamEntity::class,
+                    parentColumns = [StreamEntity.STREAM_ID],
+                    childColumns = [STREAM_ID],
+                    onDelete = ForeignKey.CASCADE, onUpdate = ForeignKey.CASCADE, deferred = true),
+            ForeignKey(
+                    entity = SubscriptionEntity::class,
+                    parentColumns = [SubscriptionEntity.SUBSCRIPTION_UID],
+                    childColumns = [SUBSCRIPTION_ID],
+                    onDelete = ForeignKey.CASCADE, onUpdate = ForeignKey.CASCADE, deferred = true)
+        ]
+)
+data class FeedEntity(
+        @ColumnInfo(name = STREAM_ID)
+        var streamId: Long,
+
+        @ColumnInfo(name = SUBSCRIPTION_ID)
+        var subscriptionId: Long
+) {
+
+    companion object {
+        const val FEED_TABLE = "feed"
+
+        const val STREAM_ID = "stream_id"
+        const val SUBSCRIPTION_ID = "subscription_id"
+    }
+}

app/src/main/java/org/schabi/newpipe/database/feed/model/FeedGroupEntity.kt

@@ -0,0 +1,39 @@
+package org.schabi.newpipe.database.feed.model
+
+import androidx.room.ColumnInfo
+import androidx.room.Entity
+import androidx.room.Index
+import androidx.room.PrimaryKey
+import org.schabi.newpipe.database.feed.model.FeedGroupEntity.Companion.FEED_GROUP_TABLE
+import org.schabi.newpipe.database.feed.model.FeedGroupEntity.Companion.SORT_ORDER
+import org.schabi.newpipe.local.subscription.FeedGroupIcon
+
+@Entity(
+        tableName = FEED_GROUP_TABLE,
+        indices = [Index(SORT_ORDER)]
+)
+data class FeedGroupEntity(
+        @PrimaryKey(autoGenerate = true)
+        @ColumnInfo(name = ID)
+        val uid: Long,
+
+        @ColumnInfo(name = NAME)
+        var name: String,
+
+        @ColumnInfo(name = ICON)
+        var icon: FeedGroupIcon,
+
+        @ColumnInfo(name = SORT_ORDER)
+        var sortOrder: Long = -1
+) {
+    companion object {
+        const val FEED_GROUP_TABLE = "feed_group"
+
+        const val ID = "uid"
+        const val NAME = "name"
+        const val ICON = "icon_id"
+        const val SORT_ORDER = "sort_order"
+
+        const val GROUP_ALL_ID = -1L
+    }
+}

app/src/main/java/org/schabi/newpipe/database/feed/model/FeedGroupSubscriptionEntity.kt

@@ -0,0 +1,45 @@
+package org.schabi.newpipe.database.feed.model
+
+import androidx.room.ColumnInfo
+import androidx.room.Entity
+import androidx.room.ForeignKey
+import androidx.room.ForeignKey.CASCADE
+import androidx.room.Index
+import org.schabi.newpipe.database.feed.model.FeedGroupSubscriptionEntity.Companion.FEED_GROUP_SUBSCRIPTION_TABLE
+import org.schabi.newpipe.database.feed.model.FeedGroupSubscriptionEntity.Companion.GROUP_ID
+import org.schabi.newpipe.database.feed.model.FeedGroupSubscriptionEntity.Companion.SUBSCRIPTION_ID
+import org.schabi.newpipe.database.subscription.SubscriptionEntity
+
+@Entity(
+        tableName = FEED_GROUP_SUBSCRIPTION_TABLE,
+        primaryKeys = [GROUP_ID, SUBSCRIPTION_ID],
+        indices = [Index(SUBSCRIPTION_ID)],
+        foreignKeys = [
+            ForeignKey(
+                    entity = FeedGroupEntity::class,
+                    parentColumns = [FeedGroupEntity.ID],
+                    childColumns = [GROUP_ID],
+                    onDelete = CASCADE, onUpdate = CASCADE, deferred = true),
+
+            ForeignKey(
+                    entity = SubscriptionEntity::class,
+                    parentColumns = [SubscriptionEntity.SUBSCRIPTION_UID],
+                    childColumns = [SUBSCRIPTION_ID],
+                    onDelete = CASCADE, onUpdate = CASCADE, deferred = true)
+        ]
+)
+data class FeedGroupSubscriptionEntity(
+        @ColumnInfo(name = GROUP_ID)
+        var feedGroupId: Long,
+
+        @ColumnInfo(name = SUBSCRIPTION_ID)
+        var subscriptionId: Long
+) {
+
+    companion object {
+        const val FEED_GROUP_SUBSCRIPTION_TABLE = "feed_group_subscription_join"
+
+        const val GROUP_ID = "group_id"
+        const val SUBSCRIPTION_ID = "subscription_id"
+    }
+}

app/src/main/java/org/schabi/newpipe/database/feed/model/FeedLastUpdatedEntity.kt

@@ -0,0 +1,37 @@
+package org.schabi.newpipe.database.feed.model
+
+import androidx.room.ColumnInfo
+import androidx.room.Entity
+import androidx.room.ForeignKey
+import androidx.room.PrimaryKey
+import org.schabi.newpipe.database.feed.model.FeedLastUpdatedEntity.Companion.FEED_LAST_UPDATED_TABLE
+import org.schabi.newpipe.database.feed.model.FeedLastUpdatedEntity.Companion.SUBSCRIPTION_ID
+import org.schabi.newpipe.database.subscription.SubscriptionEntity
+import java.util.*
+
+@Entity(
+        tableName = FEED_LAST_UPDATED_TABLE,
+        foreignKeys = [
+            ForeignKey(
+                    entity = SubscriptionEntity::class,
+                    parentColumns = [SubscriptionEntity.SUBSCRIPTION_UID],
+                    childColumns = [SUBSCRIPTION_ID],
+                    onDelete = ForeignKey.CASCADE, onUpdate = ForeignKey.CASCADE, deferred = true)
+        ]
+)
+data class FeedLastUpdatedEntity(
+        @PrimaryKey
+        @ColumnInfo(name = SUBSCRIPTION_ID)
+        var subscriptionId: Long,
+
+        @ColumnInfo(name = LAST_UPDATED)
+        var lastUpdated: Date? = null
+) {
+
+    companion object {
+        const val FEED_LAST_UPDATED_TABLE = "feed_last_updated"
+
+        const val SUBSCRIPTION_ID = "subscription_id"
+        const val LAST_UPDATED = "last_updated"
+    }
+}

app/src/main/java/org/schabi/newpipe/database/history/model/StreamHistoryEntry.java

@@ -1,59 +0,0 @@
-package org.schabi.newpipe.database.history.model;
-
-import androidx.room.ColumnInfo;
-
-import org.schabi.newpipe.database.stream.model.StreamEntity;
-import org.schabi.newpipe.extractor.stream.StreamType;
-
-import java.util.Date;
-
-public class StreamHistoryEntry {
-    @ColumnInfo(name = StreamEntity.STREAM_ID)
-    final public long uid;
-    @ColumnInfo(name = StreamEntity.STREAM_SERVICE_ID)
-    final public int serviceId;
-    @ColumnInfo(name = StreamEntity.STREAM_URL)
-    final public String url;
-    @ColumnInfo(name = StreamEntity.STREAM_TITLE)
-    final public String title;
-    @ColumnInfo(name = StreamEntity.STREAM_TYPE)
-    final public StreamType streamType;
-    @ColumnInfo(name = StreamEntity.STREAM_DURATION)
-    final public long duration;
-    @ColumnInfo(name = StreamEntity.STREAM_UPLOADER)
-    final public String uploader;
-    @ColumnInfo(name = StreamEntity.STREAM_THUMBNAIL_URL)
-    final public String thumbnailUrl;
-    @ColumnInfo(name = StreamHistoryEntity.JOIN_STREAM_ID)
-    final public long streamId;
-    @ColumnInfo(name = StreamHistoryEntity.STREAM_ACCESS_DATE)
-    final public Date accessDate;
-    @ColumnInfo(name = StreamHistoryEntity.STREAM_REPEAT_COUNT)
-    final public long repeatCount;
-
-    public StreamHistoryEntry(long uid, int serviceId, String url, String title,
-                              StreamType streamType, long duration, String uploader,
-                              String thumbnailUrl, long streamId, Date accessDate,
-                              long repeatCount) {
-        this.uid = uid;
-        this.serviceId = serviceId;
-        this.url = url;
-        this.title = title;
-        this.streamType = streamType;
-        this.duration = duration;
-        this.uploader = uploader;
-        this.thumbnailUrl = thumbnailUrl;
-        this.streamId = streamId;
-        this.accessDate = accessDate;
-        this.repeatCount = repeatCount;
-    }
-
-    public StreamHistoryEntity toStreamHistoryEntity() {
-        return new StreamHistoryEntity(streamId, accessDate, repeatCount);
-    }
-
-    public boolean hasEqualValues(StreamHistoryEntry other) {
-        return this.uid == other.uid && streamId == other.streamId &&
-                accessDate.compareTo(other.accessDate) == 0;
-    }
-}

app/src/main/java/org/schabi/newpipe/database/history/model/StreamHistoryEntry.kt

@@ -0,0 +1,30 @@
+package org.schabi.newpipe.database.history.model
+
+import androidx.room.ColumnInfo
+import androidx.room.Embedded
+import org.schabi.newpipe.database.stream.model.StreamEntity
+import java.util.*
+
+data class StreamHistoryEntry(
+        @Embedded
+        val streamEntity: StreamEntity,
+
+        @ColumnInfo(name = StreamHistoryEntity.JOIN_STREAM_ID)
+        val streamId: Long,
+
+        @ColumnInfo(name = StreamHistoryEntity.STREAM_ACCESS_DATE)
+        val accessDate: Date,
+
+        @ColumnInfo(name = StreamHistoryEntity.STREAM_REPEAT_COUNT)
+        val repeatCount: Long
+) {
+
+    fun toStreamHistoryEntity(): StreamHistoryEntity {
+        return StreamHistoryEntity(streamId, accessDate, repeatCount)
+    }
+
+    fun hasEqualValues(other: StreamHistoryEntry): Boolean {
+        return this.streamEntity.uid == other.streamEntity.uid && streamId == other.streamId &&
+                accessDate.compareTo(other.accessDate) == 0
+    }
+}

app/src/main/java/org/schabi/newpipe/database/playlist/PlaylistStreamEntry.java

@@ -1,60 +0,0 @@
-package org.schabi.newpipe.database.playlist;
-
-import androidx.room.ColumnInfo;
-
-import org.schabi.newpipe.database.LocalItem;
-import org.schabi.newpipe.database.playlist.model.PlaylistStreamEntity;
-import org.schabi.newpipe.database.stream.model.StreamEntity;
-import org.schabi.newpipe.extractor.stream.StreamInfoItem;
-import org.schabi.newpipe.extractor.stream.StreamType;
-
-public class PlaylistStreamEntry implements LocalItem {
-    @ColumnInfo(name = StreamEntity.STREAM_ID)
-    final public long uid;
-    @ColumnInfo(name = StreamEntity.STREAM_SERVICE_ID)
-    final public int serviceId;
-    @ColumnInfo(name = StreamEntity.STREAM_URL)
-    final public String url;
-    @ColumnInfo(name = StreamEntity.STREAM_TITLE)
-    final public String title;
-    @ColumnInfo(name = StreamEntity.STREAM_TYPE)
-    final public StreamType streamType;
-    @ColumnInfo(name = StreamEntity.STREAM_DURATION)
-    final public long duration;
-    @ColumnInfo(name = StreamEntity.STREAM_UPLOADER)
-    final public String uploader;
-    @ColumnInfo(name = StreamEntity.STREAM_THUMBNAIL_URL)
-    final public String thumbnailUrl;
-    @ColumnInfo(name = PlaylistStreamEntity.JOIN_STREAM_ID)
-    final public long streamId;
-    @ColumnInfo(name = PlaylistStreamEntity.JOIN_INDEX)
-    final public int joinIndex;
-
-    public PlaylistStreamEntry(long uid, int serviceId, String url, String title,
-                               StreamType streamType, long duration, String uploader,
-                               String thumbnailUrl, long streamId, int joinIndex) {
-        this.uid = uid;
-        this.serviceId = serviceId;
-        this.url = url;
-        this.title = title;
-        this.streamType = streamType;
-        this.duration = duration;
-        this.uploader = uploader;
-        this.thumbnailUrl = thumbnailUrl;
-        this.streamId = streamId;
-        this.joinIndex = joinIndex;
-    }
-
-    public StreamInfoItem toStreamInfoItem() throws IllegalArgumentException {
-        StreamInfoItem item = new StreamInfoItem(serviceId, url, title, streamType);
-        item.setThumbnailUrl(thumbnailUrl);
-        item.setUploaderName(uploader);
-        item.setDuration(duration);
-        return item;
-    }
-
-    @Override
-    public LocalItemType getLocalItemType() {
-        return LocalItemType.PLAYLIST_STREAM_ITEM;
-    }
-}

app/src/main/java/org/schabi/newpipe/database/playlist/PlaylistStreamEntry.kt

@@ -0,0 +1,34 @@
+package org.schabi.newpipe.database.playlist
+
+import androidx.room.ColumnInfo
+import androidx.room.Embedded
+import org.schabi.newpipe.database.LocalItem
+import org.schabi.newpipe.database.playlist.model.PlaylistStreamEntity
+import org.schabi.newpipe.database.stream.model.StreamEntity
+import org.schabi.newpipe.extractor.stream.StreamInfoItem
+
+class PlaylistStreamEntry(
+        @Embedded
+        val streamEntity: StreamEntity,
+
+        @ColumnInfo(name = PlaylistStreamEntity.JOIN_STREAM_ID)
+        val streamId: Long,
+
+        @ColumnInfo(name = PlaylistStreamEntity.JOIN_INDEX)
+        val joinIndex: Int
+) : LocalItem {
+
+    @Throws(IllegalArgumentException::class)
+    fun toStreamInfoItem(): StreamInfoItem {
+        val item = StreamInfoItem(streamEntity.serviceId, streamEntity.url, streamEntity.title, streamEntity.streamType)
+        item.duration = streamEntity.duration
+        item.uploaderName = streamEntity.uploader
+        item.thumbnailUrl = streamEntity.thumbnailUrl
+
+        return item
+    }
+
+    override fun getLocalItemType(): LocalItem.LocalItemType {
+        return LocalItem.LocalItemType.PLAYLIST_STREAM_ITEM
+    }
+}

app/src/main/java/org/schabi/newpipe/database/stream/StreamStatisticsEntry.java

@@ -1,69 +0,0 @@
-package org.schabi.newpipe.database.stream;
-
-import androidx.room.ColumnInfo;
-
-import org.schabi.newpipe.database.LocalItem;
-import org.schabi.newpipe.database.history.model.StreamHistoryEntity;
-import org.schabi.newpipe.database.stream.model.StreamEntity;
-import org.schabi.newpipe.extractor.stream.StreamInfoItem;
-import org.schabi.newpipe.extractor.stream.StreamType;
-
-import java.util.Date;
-
-public class StreamStatisticsEntry implements LocalItem {
-    final public static String STREAM_LATEST_DATE = "latestAccess";
-    final public static String STREAM_WATCH_COUNT = "watchCount";
-
-    @ColumnInfo(name = StreamEntity.STREAM_ID)
-    final public long uid;
-    @ColumnInfo(name = StreamEntity.STREAM_SERVICE_ID)
-    final public int serviceId;
-    @ColumnInfo(name = StreamEntity.STREAM_URL)
-    final public String url;
-    @ColumnInfo(name = StreamEntity.STREAM_TITLE)
-    final public String title;
-    @ColumnInfo(name = StreamEntity.STREAM_TYPE)
-    final public StreamType streamType;
-    @ColumnInfo(name = StreamEntity.STREAM_DURATION)
-    final public long duration;
-    @ColumnInfo(name = StreamEntity.STREAM_UPLOADER)
-    final public String uploader;
-    @ColumnInfo(name = StreamEntity.STREAM_THUMBNAIL_URL)
-    final public String thumbnailUrl;
-    @ColumnInfo(name = StreamHistoryEntity.JOIN_STREAM_ID)
-    final public long streamId;
-    @ColumnInfo(name = StreamStatisticsEntry.STREAM_LATEST_DATE)
-    final public Date latestAccessDate;
-    @ColumnInfo(name = StreamStatisticsEntry.STREAM_WATCH_COUNT)
-    final public long watchCount;
-
-    public StreamStatisticsEntry(long uid, int serviceId, String url, String title,
-                                 StreamType streamType, long duration, String uploader,
-                                 String thumbnailUrl, long streamId, Date latestAccessDate,
-                                 long watchCount) {
-        this.uid = uid;
-        this.serviceId = serviceId;
-        this.url = url;
-        this.title = title;
-        this.streamType = streamType;
-        this.duration = duration;
-        this.uploader = uploader;
-        this.thumbnailUrl = thumbnailUrl;
-        this.streamId = streamId;
-        this.latestAccessDate = latestAccessDate;
-        this.watchCount = watchCount;
-    }
-
-    public StreamInfoItem toStreamInfoItem() {
-        StreamInfoItem item = new StreamInfoItem(serviceId, url, title, streamType);
-        item.setDuration(duration);
-        item.setUploaderName(uploader);
-        item.setThumbnailUrl(thumbnailUrl);
-        return item;
-    }
-
-    @Override
-    public LocalItemType getLocalItemType() {
-        return LocalItemType.STATISTIC_STREAM_ITEM;
-    }
-}

app/src/main/java/org/schabi/newpipe/database/stream/StreamStatisticsEntry.kt

@@ -0,0 +1,42 @@
+package org.schabi.newpipe.database.stream
+
+import androidx.room.ColumnInfo
+import androidx.room.Embedded
+import org.schabi.newpipe.database.LocalItem
+import org.schabi.newpipe.database.history.model.StreamHistoryEntity
+import org.schabi.newpipe.database.stream.model.StreamEntity
+import org.schabi.newpipe.extractor.stream.StreamInfoItem
+import java.util.*
+
+class StreamStatisticsEntry(
+        @Embedded
+        val streamEntity: StreamEntity,
+
+        @ColumnInfo(name = StreamHistoryEntity.JOIN_STREAM_ID)
+        val streamId: Long,
+
+        @ColumnInfo(name = STREAM_LATEST_DATE)
+        val latestAccessDate: Date,
+
+        @ColumnInfo(name = STREAM_WATCH_COUNT)
+        val watchCount: Long
+) : LocalItem {
+
+    fun toStreamInfoItem(): StreamInfoItem {
+        val item = StreamInfoItem(streamEntity.serviceId, streamEntity.url, streamEntity.title, streamEntity.streamType)
+        item.duration = streamEntity.duration
+        item.uploaderName = streamEntity.uploader
+        item.thumbnailUrl = streamEntity.thumbnailUrl
+
+        return item
+    }
+
+    override fun getLocalItemType(): LocalItem.LocalItemType {
+        return LocalItem.LocalItemType.STATISTIC_STREAM_ITEM
+    }
+
+    companion object {
+        const val STREAM_LATEST_DATE = "latestAccess"
+        const val STREAM_WATCH_COUNT = "watchCount"
+    }
+}

app/src/main/java/org/schabi/newpipe/database/stream/dao/StreamDAO.java

@@ -1,98 +0,0 @@
-package org.schabi.newpipe.database.stream.dao;
-
-import androidx.room.Dao;
-import androidx.room.Insert;
-import androidx.room.OnConflictStrategy;
-import androidx.room.Query;
-import androidx.room.Transaction;
-
-import org.schabi.newpipe.database.BasicDAO;
-import org.schabi.newpipe.database.playlist.model.PlaylistStreamEntity;
-import org.schabi.newpipe.database.stream.model.StreamEntity;
-import org.schabi.newpipe.database.history.model.StreamHistoryEntity;
-
-import java.util.ArrayList;
-import java.util.List;
-
-import io.reactivex.Flowable;
-
-import static org.schabi.newpipe.database.playlist.model.PlaylistStreamEntity.PLAYLIST_STREAM_JOIN_TABLE;
-import static org.schabi.newpipe.database.stream.model.StreamEntity.STREAM_ID;
-import static org.schabi.newpipe.database.stream.model.StreamEntity.STREAM_SERVICE_ID;
-import static org.schabi.newpipe.database.stream.model.StreamEntity.STREAM_TABLE;
-import static org.schabi.newpipe.database.stream.model.StreamEntity.STREAM_URL;
-import static org.schabi.newpipe.database.history.model.StreamHistoryEntity.STREAM_HISTORY_TABLE;
-
-@Dao
-public abstract class StreamDAO implements BasicDAO<StreamEntity> {
-    @Override
-    @Query("SELECT * FROM " + STREAM_TABLE)
-    public abstract Flowable<List<StreamEntity>> getAll();
-
-    @Override
-    @Query("DELETE FROM " + STREAM_TABLE)
-    public abstract int deleteAll();
-
-    @Override
-    @Query("SELECT * FROM " + STREAM_TABLE + " WHERE " + STREAM_SERVICE_ID + " = :serviceId")
-    public abstract Flowable<List<StreamEntity>> listByService(int serviceId);
-
-    @Query("SELECT * FROM " + STREAM_TABLE + " WHERE " +
-            STREAM_URL + " = :url AND " +
-            STREAM_SERVICE_ID + " = :serviceId")
-    public abstract Flowable<List<StreamEntity>> getStream(long serviceId, String url);
-
-    @Insert(onConflict = OnConflictStrategy.IGNORE)
-    abstract void silentInsertAllInternal(final List<StreamEntity> streams);
-
-    @Query("SELECT " + STREAM_ID + " FROM " + STREAM_TABLE + " WHERE " +
-            STREAM_URL + " = :url AND " +
-            STREAM_SERVICE_ID + " = :serviceId")
-    abstract Long getStreamIdInternal(long serviceId, String url);
-
-    @Transaction
-    public long upsert(StreamEntity stream) {
-        final Long streamIdCandidate = getStreamIdInternal(stream.getServiceId(), stream.getUrl());
-
-        if (streamIdCandidate == null) {
-            return insert(stream);
-        } else {
-            stream.setUid(streamIdCandidate);
-            update(stream);
-            return streamIdCandidate;
-        }
-    }
-
-    @Transaction
-    public List<Long> upsertAll(List<StreamEntity> streams) {
-        silentInsertAllInternal(streams);
-
-        final List<Long> streamIds = new ArrayList<>(streams.size());
-        for (StreamEntity stream : streams) {
-            final Long streamId = getStreamIdInternal(stream.getServiceId(), stream.getUrl());
-            if (streamId == null) {
-                throw new IllegalStateException("StreamID cannot be null just after insertion.");
-            }
-
-            streamIds.add(streamId);
-            stream.setUid(streamId);
-        }
-
-        update(streams);
-        return streamIds;
-    }
-
-    @Query("DELETE FROM " + STREAM_TABLE + " WHERE " + STREAM_ID +
-            " NOT IN " +
-            "(SELECT DISTINCT " + STREAM_ID + " FROM " + STREAM_TABLE +
-
-            " LEFT JOIN " + STREAM_HISTORY_TABLE +
-            " ON " + STREAM_ID + " = " +
-            StreamHistoryEntity.STREAM_HISTORY_TABLE + "." + StreamHistoryEntity.JOIN_STREAM_ID +
-
-            " LEFT JOIN " + PLAYLIST_STREAM_JOIN_TABLE +
-            " ON " + STREAM_ID + " = " +
-            PlaylistStreamEntity.PLAYLIST_STREAM_JOIN_TABLE + "." + PlaylistStreamEntity.JOIN_STREAM_ID +
-            ")")
-    public abstract int deleteOrphans();
-}