Merge pull request #3267 from TeamNewPipe/release_0.19.0

Release 0.19.0

.github/ISSUE_TEMPLATE.md

@@ -1,3 +0,0 @@
-- [ ] I carefully read the [contribution guidelines](https://github.com/TeamNewPipe/NewPipe/blob/HEAD/.github/CONTRIBUTING.md) and agree to them.
-- [ ] I checked if the issue/feature exists in the latest version.
-- [ ] I did use the [incredible bugreport to markdown converter](https://teamnewpipe.github.io/CrashReportToMarkdown/) to paste bug reports.

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,44 @@
+---
+name: Bug report
+about: Create a bug report to help us improve
+labels: bug
+assignees: ''
+
+---
+
+<!--
+Oh no, a bug! It happens. Thanks for reporting an issue with NewPipe.
+
+Use this template to notify us if you found a bug.
+
+To make it easier for us to help you please enter detailed information below.
+
+Please note, we only support the latest version of NewPipe and the master branch. Make sure you have that version installed. If you don't, upgrade & reproduce the problem before opening the issue. The release page (https://github.com/TeamNewPipe/NewPipe/releases/latest) is the go-to place to get this version. In order to check your app version, open the left drawer and click on "About".
+
+P.S.: Our contribution guidelines might be a nice document to read before you fill out the report :) You can find it at https://github.com/TeamNewPipe/NewPipe/blob/HEAD/.github/CONTRIBUTING.md
+-->
+### Version
+<!-- Which version are you using? -->
+- 
+
+
+### Steps to reproduce the bug
+<!-- If you can't reproduce it, please try to give as many details as possible on how you think you got to the bug. -->
+Steps to reproduce the behavior:
+1. Go to '...'
+2. Press on '....'
+3. Swipe down to '....'
+
+### Expected behavior
+Tell us what you expected to happen.
+
+### Actual behaviour
+Tell us what happens instead.
+
+### Screenshots/Screen records
+If applicable, add screenshots or a screen recording to help explain your problem. GitHub should support uploading them directly in the issue field. If your file is too big, feel free to paste a link from an image/video hoster here instead.
+
+### Logs
+If your bug includes a crash, please head over to the [incredible bugreport to markdown converter](https://teamnewpipe.github.io/CrashReportToMarkdown/). Copy the result. Paste it here:
+
+<!-- That's right, here! -->

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,28 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+labels: enhancement
+assignees: ''
+
+---
+<!-- Hey. Our contribution guidelines (https://github.com/TeamNewPipe/NewPipe/blob/HEAD/.github/CONTRIBUTING.md) might be an appropriate
+ document to read before you fill out the request :) -->
+#### Is your feature request related to a problem? Please describe it
+A clear and concise description of what the problem is.
+Example: *I want to do X, but there is no way to do it.*
+
+#### Describe the solution you'd like
+A clear and concise description of what you want to happen.
+Example: *I think it would be nice if you add feature Y which makes X possible.*
+
+#### (Optional) Describe alternatives you've considered
+A clear and concise description of any alternative solutions or features you've considered.
+Example: *I considered Z, but that didn't turn out to be a good idea because...*
+
+#### Additional context
+Add any other context or screenshots about the feature request here.
+Example: *Here's a photo of my cat!*
+
+#### How will you/everyone benefit from this feature?
+Convince us! How does it change your NewPipe experience and/or your life?
+The better this paragraph is, the more likely a developer will think about working on it.

.github/PULL_REQUEST_TEMPLATE.md

@@ -1 +1,26 @@
+<!-- Hey there. Thank you so much for improving NewPipe. Please take a moment to fill out the following suggestion on how to structure this PR description. Having roughly the same layout helps everyone considerably :)-->
+
+#### What is it?
+- [ ] Bug fix
+- [ ] Feature
+
+#### Long description of the changes in your PR
+<!-- While bullet points are the norm in this section, feel free to write a text instead if you can't fit it in a list -->
+- record videos
+- create clones
+- take over the world
+
+#### Fixes the following issue(s)
+<!-- Also add reddit or other links which are relevant to your change. -->
+- 
+
+#### Relies on the following changes
+<!-- Delete this if it doesn't apply to you. -->
+- 
+
+#### Testing apk
+<!-- Ensure that you have your changes on a new branch which has a meaningful name. This name will be used as a suffix for the app ID to allow installing and testing multiple versions of NewPipe. Do NOT name your branches like "patch-0" and "feature-1". For example, if your PR implements a bug fix for comments, an appropriate branch name would be "commentfix". -->
+debug.zip
+
+#### Agreement
 - [ ] I carefully read the [contribution guidelines](https://github.com/TeamNewPipe/NewPipe/blob/HEAD/.github/CONTRIBUTING.md) and agree to them.

README.md

@@ -13,7 +13,7 @@
 </p>
 <hr>
 <p align="center"><a href="#screenshots">Screenshots</a> &bull; <a href="#description">Description</a> &bull; <a href="#features">Features</a> &bull; <a href="#updates">Updates</a> &bull; <a href="#contribution">Contribution</a> &bull; <a href="#donate">Donate</a> &bull; <a href="#license">License</a></p>
-<p align="center"><a href="https://newpipe.schabi.org">Website</a> &bull; <a href="https://newpipe.schabi.org/blog/">Blog</a>  &bull; <a href="https://newpipe.schabi.org/press/">Press</a></p>
+<p align="center"><a href="https://newpipe.schabi.org">Website</a> &bull; <a href="https://newpipe.schabi.org/blog/">Blog</a> &bull; <a href="https://newpipe.schabi.org/FAQ/">FAQ</a> &bull; <a href="https://newpipe.schabi.org/press/">Press</a></p>
 <hr>
 
 <b>WARNING: THIS IS A BETA VERSION, THEREFORE YOU MAY ENCOUNTER BUGS. IF YOU DO, OPEN AN ISSUE VIA OUR GITHUB REPOSITORY.</b>
@@ -81,6 +81,7 @@ NewPipe supports multiple services. Our [docs](https://teamnewpipe.github.io/doc
 * YouTube
 * SoundCloud \[beta\]
 * media.ccc.de \[beta\]
+* PeerTube instances \[beta\]
 
 ## Updates
 When a change to the NewPipe code occurs (due to either adding features or bug fixing), eventually a release will occur. These are in the format x.xx.x . In order to get this new version, you can:

app/.gitignore

@@ -1,3 +1,3 @@
 .gitignore
 /build
-app.iml
+*.iml

app/build.gradle

@@ -1,4 +1,7 @@
 apply plugin: 'com.android.application'
+apply plugin: 'kotlin-android'
+apply plugin: 'kotlin-android-extensions'
+apply plugin: 'kotlin-kapt'
 
 android {
     compileSdkVersion 28
@@ -6,13 +9,20 @@ android {
 
     defaultConfig {
         applicationId "org.schabi.newpipe"
+        resValue "string", "app_name", "NewPipe"
         minSdkVersion 19
         targetSdkVersion 28
-        versionCode 790
-        versionName "0.17.4"
+        versionCode 900
+        versionName "0.19.0"
 
         testInstrumentationRunner "androidx.test.runner.AndroidJUnitRunner"
         vectorDrawables.useSupportLibrary = true
+
+        javaCompileOptions {
+            annotationProcessorOptions {
+                arguments = ["room.schemaLocation": "$projectDir/schemas".toString()]
+            }
+        }
     }
 
     buildTypes {
@@ -25,7 +35,18 @@ android {
         debug {
             multiDexEnabled true
             debuggable true
-            applicationIdSuffix ".debug"
+
+            // suffix the app id and the app name with git branch name
+            def workingBranch = getGitWorkingBranch()
+            def normalizedWorkingBranch = workingBranch.replaceAll("[^A-Za-z]+", "").toLowerCase()
+            if (normalizedWorkingBranch.isEmpty() || workingBranch == "master" || workingBranch == "dev") {
+                // default values when branch name could not be determined or is master or dev
+                applicationIdSuffix ".debug"
+                resValue "string", "app_name", "NewPipe Debug"
+            } else {
+                applicationIdSuffix ".debug." + normalizedWorkingBranch
+                resValue "string", "app_name", "NewPipe " + workingBranch
+            }
         }
     }
 
@@ -40,24 +61,38 @@ android {
         sourceCompatibility JavaVersion.VERSION_1_8
         targetCompatibility JavaVersion.VERSION_1_8
     }
+
+    // Required and used only by groupie
+    androidExtensions {
+        experimental = true
+    }
+
+    sourceSets {
+        androidTest.assets.srcDirs += files("$projectDir/schemas".toString())
+    }
 }
 
 ext {
     androidxLibVersion = '1.0.0'
-    exoPlayerLibVersion = '2.10.6'
+    exoPlayerLibVersion = '2.10.8'
     roomDbLibVersion = '2.1.0'
     leakCanaryLibVersion = '1.5.4' //1.6.1
     okHttpLibVersion = '3.12.6'
     icepickLibVersion = '3.2.0'
     stethoLibVersion = '1.5.0'
+    markwonVersion = '4.2.1'
 }
 
 dependencies {
+    implementation "org.jetbrains.kotlin:kotlin-stdlib-jdk7:$kotlin_version"
+
+    androidTestImplementation 'androidx.test.ext:junit:1.1.1'
+    androidTestImplementation "android.arch.persistence.room:testing:1.1.1"
     androidTestImplementation('androidx.test.espresso:espresso-core:3.1.0', {
         exclude module: 'support-annotations'
     })
 
-    implementation 'com.github.teamnewpipe:NewPipeExtractor:v0.17.4'
+    implementation 'com.github.TeamNewPipe:NewPipeExtractor:69e0624e3'
     testImplementation 'junit:junit:4.12'
     testImplementation 'org.mockito:mockito-core:2.23.0'
 
@@ -69,6 +104,13 @@ dependencies {
     implementation "androidx.cardview:cardview:${androidxLibVersion}"
     implementation 'androidx.constraintlayout:constraintlayout:1.1.3'
 
+    implementation 'com.xwray:groupie:2.7.0'
+    implementation 'com.xwray:groupie-kotlin-android-extensions:2.7.0'
+
+    implementation 'androidx.lifecycle:lifecycle-livedata:2.0.0'
+    implementation 'androidx.lifecycle:lifecycle-viewmodel:2.0.0'
+    implementation 'androidx.lifecycle:lifecycle-extensions:2.0.0'
+
     // Originally in NewPipeExtractor
     implementation 'com.grack:nanojson:1.1'
     implementation 'org.jsoup:jsoup:1.9.2'
@@ -89,17 +131,37 @@ dependencies {
     implementation 'io.reactivex.rxjava2:rxjava:2.2.2'
     implementation 'io.reactivex.rxjava2:rxandroid:2.1.0'
     implementation 'com.jakewharton.rxbinding2:rxbinding:2.1.1'
+    implementation 'org.ocpsoft.prettytime:prettytime:4.0.3.Final'
 
     implementation "androidx.room:room-runtime:${roomDbLibVersion}"
     implementation "androidx.room:room-rxjava2:${roomDbLibVersion}"
-    annotationProcessor "androidx.room:room-compiler:${roomDbLibVersion}"
+    kapt "androidx.room:room-compiler:${roomDbLibVersion}"
 
     implementation "frankiesardo:icepick:${icepickLibVersion}"
-    annotationProcessor "frankiesardo:icepick-processor:${icepickLibVersion}"
+    kapt "frankiesardo:icepick-processor:${icepickLibVersion}"
 
     debugImplementation "com.squareup.leakcanary:leakcanary-android:${leakCanaryLibVersion}"
     releaseImplementation "com.squareup.leakcanary:leakcanary-android-no-op:${leakCanaryLibVersion}"
 
     implementation "com.squareup.okhttp3:okhttp:${okHttpLibVersion}"
     debugImplementation "com.facebook.stetho:stetho-okhttp3:${stethoLibVersion}"
+
+    implementation "io.noties.markwon:core:${markwonVersion}"
+    implementation "io.noties.markwon:linkify:${markwonVersion}"
 }
+
+static String getGitWorkingBranch() {
+    try {
+        def gitProcess = "git rev-parse --abbrev-ref HEAD".execute()
+        gitProcess.waitFor()
+        if (gitProcess.exitValue() == 0) {
+            return gitProcess.text.trim()
+        } else {
+            // not a git repository
+            return ""
+        }
+    } catch (IOException ignored) {
+        // git was not found
+        return ""
+    }
+}

app/proguard-rules.pro

@@ -17,6 +17,9 @@
 #}
 
 -dontobfuscate
+-keep class org.schabi.newpipe.extractor.timeago.patterns.** { *; }
+-keep class org.ocpsoft.prettytime.i18n.** { *; }
+
 -keep class org.mozilla.javascript.** { *; }
 
 -keep class org.mozilla.classfile.ClassFileWriter

app/src/androidTest/java/org/schabi/newpipe/database/AppDatabaseTest.kt

@@ -0,0 +1,79 @@
+package org.schabi.newpipe.database
+
+import android.content.ContentValues
+import android.database.sqlite.SQLiteDatabase
+import androidx.room.Room
+import androidx.room.testing.MigrationTestHelper
+import androidx.sqlite.db.framework.FrameworkSQLiteOpenHelperFactory
+import androidx.test.core.app.ApplicationProvider
+import androidx.test.ext.junit.runners.AndroidJUnit4
+import androidx.test.platform.app.InstrumentationRegistry
+import org.junit.Assert.assertEquals
+import org.junit.Assert.assertNull
+import org.junit.Rule
+import org.junit.Test
+import org.junit.runner.RunWith
+import org.schabi.newpipe.extractor.stream.StreamType
+
+@RunWith(AndroidJUnit4::class)
+class AppDatabaseTest {
+    companion object {
+        private const val DEFAULT_SERVICE_ID = 0
+        private const val DEFAULT_URL = "https://www.youtube.com/watch?v=cDphUib5iG4"
+        private const val DEFAULT_TITLE = "Test Title"
+        private val DEFAULT_TYPE = StreamType.VIDEO_STREAM
+        private const val DEFAULT_DURATION = 480L
+        private const val DEFAULT_UPLOADER_NAME = "Uploader Test"
+        private const val DEFAULT_THUMBNAIL = "https://example.com/example.jpg"
+    }
+
+    @get:Rule val testHelper = MigrationTestHelper(InstrumentationRegistry.getInstrumentation(),
+            AppDatabase::class.java.canonicalName, FrameworkSQLiteOpenHelperFactory());
+
+    @Test
+    fun migrateDatabaseFrom2to3() {
+        val databaseInV2 = testHelper.createDatabase(AppDatabase.DATABASE_NAME, Migrations.DB_VER_2)
+
+        databaseInV2.run {
+            insert("streams", SQLiteDatabase.CONFLICT_FAIL, ContentValues().apply {
+                // put("uid", null)
+                put("service_id", DEFAULT_SERVICE_ID)
+                put("url", DEFAULT_URL)
+                put("title", DEFAULT_TITLE)
+                put("stream_type", DEFAULT_TYPE.name)
+                put("duration", DEFAULT_DURATION)
+                put("uploader", DEFAULT_UPLOADER_NAME)
+                put("thumbnail_url", DEFAULT_THUMBNAIL)
+            })
+            close()
+        }
+
+        testHelper.runMigrationsAndValidate(AppDatabase.DATABASE_NAME, Migrations.DB_VER_3,
+                true, Migrations.MIGRATION_2_3);
+
+        val migratedDatabaseV3 = getMigratedDatabase()
+        val listFromDB = migratedDatabaseV3.streamDAO().all.blockingFirst()
+        assertEquals(1, listFromDB.size)
+
+        val streamFromMigratedDatabase = listFromDB.first()
+        assertEquals(DEFAULT_SERVICE_ID, streamFromMigratedDatabase.serviceId)
+        assertEquals(DEFAULT_URL, streamFromMigratedDatabase.url)
+        assertEquals(DEFAULT_TITLE, streamFromMigratedDatabase.title)
+        assertEquals(DEFAULT_TYPE, streamFromMigratedDatabase.streamType)
+        assertEquals(DEFAULT_DURATION, streamFromMigratedDatabase.duration)
+        assertEquals(DEFAULT_UPLOADER_NAME, streamFromMigratedDatabase.uploader)
+        assertEquals(DEFAULT_THUMBNAIL, streamFromMigratedDatabase.thumbnailUrl)
+        assertNull(streamFromMigratedDatabase.viewCount)
+        assertNull(streamFromMigratedDatabase.textualUploadDate)
+        assertNull(streamFromMigratedDatabase.uploadDate)
+        assertNull(streamFromMigratedDatabase.isUploadDateApproximation)
+    }
+
+    private fun getMigratedDatabase(): AppDatabase {
+        val database: AppDatabase = Room.databaseBuilder(ApplicationProvider.getApplicationContext(),
+                AppDatabase::class.java, AppDatabase.DATABASE_NAME)
+                .build()
+        testHelper.closeWhenFinished(database)
+        return database
+    }
+}

app/src/debug/AndroidManifest.xml

@@ -6,12 +6,5 @@
 
     <application
         android:name=".DebugApp"
-        android:label="NewPipe Debug"
-        tools:replace="android:name, android:label">
-        <activity
-            android:name=".MainActivity"
-            android:label="NewPipe Debug"
-            tools:replace="android:label"/>
-    </application>
-
+        tools:replace="android:name" />
 </manifest>

app/src/debug/java/org/schabi/newpipe/DebugApp.java

@@ -15,7 +15,7 @@ import com.squareup.leakcanary.LeakCanary;
 import com.squareup.leakcanary.LeakDirectoryProvider;
 import com.squareup.leakcanary.RefWatcher;
 
-import org.schabi.newpipe.extractor.Downloader;
+import org.schabi.newpipe.extractor.downloader.Downloader;
 
 import java.io.File;
 import java.util.concurrent.TimeUnit;
@@ -39,7 +39,7 @@ public class DebugApp extends App {
 
     @Override
     protected Downloader getDownloader() {
-        return org.schabi.newpipe.Downloader.init(new OkHttpClient.Builder()
+        return DownloaderImpl.init(new OkHttpClient.Builder()
                 .addNetworkInterceptor(new StethoInterceptor()));
     }
 

app/src/main/AndroidManifest.xml

@@ -61,7 +61,8 @@
             android:name=".player.MainVideoPlayer"
             android:configChanges="keyboard|keyboardHidden|orientation|screenSize"
             android:label="@string/app_name"
-            android:launchMode="singleTask"/>
+            android:launchMode="singleTask"
+            android:theme="@style/VideoPlayerTheme"/>
 
         <activity
             android:name=".settings.SettingsActivity"
@@ -73,6 +74,7 @@
 
         <service android:name=".local.subscription.services.SubscriptionsImportService"/>
         <service android:name=".local.subscription.services.SubscriptionsExportService"/>
+        <service android:name=".local.feed.service.FeedLoadService"/>
 
         <activity
             android:name=".PanicResponderActivity"
@@ -112,7 +114,7 @@
 
         <activity
             android:name=".ReCaptchaActivity"
-            android:label="@string/reCaptchaActivity"/>
+            android:label="@string/recaptcha"/>
 
         <provider
             android:name="androidx.core.content.FileProvider"
@@ -145,6 +147,7 @@
                 <data android:host="youtube.com"/>
                 <data android:host="m.youtube.com"/>
                 <data android:host="www.youtube.com"/>
+                <data android:host="music.youtube.com"/>
                 <!-- video prefix -->
                 <data android:pathPrefix="/v/"/>
                 <data android:pathPrefix="/embed/"/>
@@ -153,6 +156,7 @@
                 <!-- channel prefix -->
                 <data android:pathPrefix="/channel/"/>
                 <data android:pathPrefix="/user/"/>
+                <data android:pathPrefix="/c/"/>
                 <!-- playlist prefix -->
                 <data android:pathPrefix="/playlist"/>
             </intent-filter>
@@ -242,14 +246,7 @@
                 <data android:host="tube.poal.co"/>
                 <data android:host="invidious.13ad.de"/>
                 <data android:host="yt.elukerio.org"/>
-                <!-- video prefix -->
-                <data android:pathPrefix="/embed/"/>
-                <data android:pathPrefix="/watch"/>
-                <!-- channel prefix -->
-                <data android:pathPrefix="/channel/"/>
-                <data android:pathPrefix="/user/"/>
-                <!-- playlist prefix -->
-                <data android:pathPrefix="/playlist"/>
+                <data android:pathPrefix="/"/>
             </intent-filter>
 
             <!-- Soundcloud filter -->
@@ -275,8 +272,26 @@
                 <category android:name="android.intent.category.DEFAULT"/>
                 <data android:mimeType="text/plain"/>
             </intent-filter>
-        </activity>
 
+            <!-- MediaCCC filter -->
+            <intent-filter>
+                <action android:name="android.intent.action.VIEW"/>
+                <action android:name="android.media.action.MEDIA_PLAY_FROM_SEARCH"/>
+                <action android:name="android.nfc.action.NDEF_DISCOVERED"/>
+
+                <category android:name="android.intent.category.DEFAULT"/>
+                <category android:name="android.intent.category.BROWSABLE"/>
+
+                <data android:scheme="http"/>
+                <data android:scheme="https"/>
+                <data android:host="media.ccc.de"/>
+                <!-- video prefix -->
+                <data android:pathPrefix="/v/"/>
+                <!-- channel prefix-->
+                <data android:pathPrefix="/c/"/>
+                <data android:pathPrefix="/b/"/>
+            </intent-filter>
+        </activity>
         <service
             android:name=".RouterActivity$FetcherService"
             android:exported="false"/>

app/src/main/java/org/schabi/newpipe/App.java

@@ -6,9 +6,9 @@ import android.app.NotificationChannel;
 import android.app.NotificationManager;
 import android.content.Context;
 import android.os.Build;
+import android.util.Log;
 
 import androidx.annotation.Nullable;
-import android.util.Log;
 
 import com.nostra13.universalimageloader.cache.memory.impl.LRULimitedMemoryCache;
 import com.nostra13.universalimageloader.core.ImageLoader;
@@ -21,13 +21,15 @@ import org.acra.config.ACRAConfiguration;
 import org.acra.config.ACRAConfigurationException;
 import org.acra.config.ConfigurationBuilder;
 import org.acra.sender.ReportSenderFactory;
-import org.schabi.newpipe.extractor.Downloader;
 import org.schabi.newpipe.extractor.NewPipe;
+import org.schabi.newpipe.extractor.downloader.Downloader;
 import org.schabi.newpipe.report.AcraReportSenderFactory;
 import org.schabi.newpipe.report.ErrorActivity;
 import org.schabi.newpipe.report.UserAction;
 import org.schabi.newpipe.settings.SettingsActivity;
 import org.schabi.newpipe.util.ExtractorHelper;
+import org.schabi.newpipe.util.Localization;
+import org.schabi.newpipe.util.ServiceHelper;
 import org.schabi.newpipe.util.StateSaver;
 
 import java.io.IOException;
@@ -95,10 +97,15 @@ public class App extends Application {
         SettingsActivity.initSettings(this);
 
         NewPipe.init(getDownloader(),
-                org.schabi.newpipe.util.Localization.getPreferredExtractorLocal(this));
+                Localization.getPreferredLocalization(this),
+                Localization.getPreferredContentCountry(this));
+        Localization.init(getApplicationContext());
+
         StateSaver.init(this);
         initNotificationChannel();
 
+        ServiceHelper.initServices(this);
+
         // Initialize image loader
         ImageLoader.getInstance().init(getImageLoaderConfigurations(10, 50));
 
@@ -109,7 +116,7 @@ public class App extends Application {
     }
 
     protected Downloader getDownloader() {
-        return org.schabi.newpipe.Downloader.init(null);
+        return DownloaderImpl.init(null);
     }
 
     private void configureRxJavaErrorHandler() {

app/src/main/java/org/schabi/newpipe/BaseFragment.java

@@ -107,6 +107,7 @@ public abstract class BaseFragment extends Fragment {
         if (DEBUG) Log.d(TAG, "setTitle() called with: title = [" + title + "]");
         if((!useAsFrontPage || mIsVisibleToUser)
             && (activity != null && activity.getSupportActionBar() != null)) {
+            activity.getSupportActionBar().setDisplayShowTitleEnabled(true);
             activity.getSupportActionBar().setTitle(title);
         }
     }

app/src/main/java/org/schabi/newpipe/DownloaderImpl.java

@@ -0,0 +1,220 @@
+package org.schabi.newpipe;
+
+import android.os.Build;
+import android.text.TextUtils;
+
+import org.schabi.newpipe.extractor.downloader.Downloader;
+import org.schabi.newpipe.extractor.downloader.Request;
+import org.schabi.newpipe.extractor.downloader.Response;
+import org.schabi.newpipe.extractor.exceptions.ReCaptchaException;
+import org.schabi.newpipe.util.TLSSocketFactoryCompat;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.security.KeyManagementException;
+import java.security.KeyStore;
+import java.security.KeyStoreException;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+import javax.net.ssl.SSLSocketFactory;
+import javax.net.ssl.TrustManager;
+import javax.net.ssl.TrustManagerFactory;
+import javax.net.ssl.X509TrustManager;
+
+import androidx.annotation.NonNull;
+import androidx.annotation.Nullable;
+
+import okhttp3.CipherSuite;
+import okhttp3.ConnectionSpec;
+import okhttp3.OkHttpClient;
+import okhttp3.RequestBody;
+import okhttp3.ResponseBody;
+
+import static org.schabi.newpipe.MainActivity.DEBUG;
+
+public class DownloaderImpl extends Downloader {
+    public static final String USER_AGENT = "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Firefox/68.0";
+
+    private static DownloaderImpl instance;
+    private String mCookies;
+    private OkHttpClient client;
+
+    private DownloaderImpl(OkHttpClient.Builder builder) {
+        if (Build.VERSION.SDK_INT == Build.VERSION_CODES.KITKAT) {
+            enableModernTLS(builder);
+        }
+        this.client = builder
+                .readTimeout(30, TimeUnit.SECONDS)
+                //.cache(new Cache(new File(context.getExternalCacheDir(), "okhttp"), 16 * 1024 * 1024))
+                .build();
+    }
+
+    /**
+     * It's recommended to call exactly once in the entire lifetime of the application.
+     *
+     * @param builder if null, default builder will be used
+     */
+    public static DownloaderImpl init(@Nullable OkHttpClient.Builder builder) {
+        return instance = new DownloaderImpl(builder != null ? builder : new OkHttpClient.Builder());
+    }
+
+    public static DownloaderImpl getInstance() {
+        return instance;
+    }
+
+    public String getCookies() {
+        return mCookies;
+    }
+
+    public void setCookies(String cookies) {
+        mCookies = cookies;
+    }
+
+    /**
+     * Get the size of the content that the url is pointing by firing a HEAD request.
+     *
+     * @param url an url pointing to the content
+     * @return the size of the content, in bytes
+     */
+    public long getContentLength(String url) throws IOException {
+        try {
+            final Response response = head(url);
+            return Long.parseLong(response.getHeader("Content-Length"));
+        } catch (NumberFormatException e) {
+            throw new IOException("Invalid content length", e);
+        } catch (ReCaptchaException e) {
+            throw new IOException(e);
+        }
+    }
+
+    public InputStream stream(String siteUrl) throws IOException {
+        try {
+            final okhttp3.Request.Builder requestBuilder = new okhttp3.Request.Builder()
+                    .method("GET", null).url(siteUrl)
+                    .addHeader("User-Agent", USER_AGENT);
+
+            if (!TextUtils.isEmpty(mCookies)) {
+                requestBuilder.addHeader("Cookie", mCookies);
+            }
+
+            final okhttp3.Request request = requestBuilder.build();
+            final okhttp3.Response response = client.newCall(request).execute();
+            final ResponseBody body = response.body();
+
+            if (response.code() == 429) {
+                throw new ReCaptchaException("reCaptcha Challenge requested", siteUrl);
+            }
+
+            if (body == null) {
+                response.close();
+                return null;
+            }
+
+            return body.byteStream();
+        } catch (ReCaptchaException e) {
+            throw new IOException(e.getMessage(), e.getCause());
+        }
+    }
+
+    @Override
+    public Response execute(@NonNull Request request) throws IOException, ReCaptchaException {
+        final String httpMethod = request.httpMethod();
+        final String url = request.url();
+        final Map<String, List<String>> headers = request.headers();
+        final byte[] dataToSend = request.dataToSend();
+
+        RequestBody requestBody = null;
+        if (dataToSend != null) {
+            requestBody = RequestBody.create(null, dataToSend);
+        }
+
+        final okhttp3.Request.Builder requestBuilder = new okhttp3.Request.Builder()
+                .method(httpMethod, requestBody).url(url)
+                .addHeader("User-Agent", USER_AGENT);
+
+        if (!TextUtils.isEmpty(mCookies)) {
+            requestBuilder.addHeader("Cookie", mCookies);
+        }
+
+        for (Map.Entry<String, List<String>> pair : headers.entrySet()) {
+            final String headerName = pair.getKey();
+            final List<String> headerValueList = pair.getValue();
+
+            if (headerValueList.size() > 1) {
+                requestBuilder.removeHeader(headerName);
+                for (String headerValue : headerValueList) {
+                    requestBuilder.addHeader(headerName, headerValue);
+                }
+            } else if (headerValueList.size() == 1) {
+                requestBuilder.header(headerName, headerValueList.get(0));
+            }
+
+        }
+
+        final okhttp3.Response response = client.newCall(requestBuilder.build()).execute();
+
+        if (response.code() == 429) {
+            response.close();
+
+            throw new ReCaptchaException("reCaptcha Challenge requested", url);
+        }
+
+        final ResponseBody body = response.body();
+        String responseBodyToReturn = null;
+
+        if (body != null) {
+            responseBodyToReturn = body.string();
+        }
+
+        final String latestUrl = response.request().url().toString();
+        return new Response(response.code(), response.message(), response.headers().toMultimap(), responseBodyToReturn, latestUrl);
+    }
+
+    /**
+     * Enable TLS 1.2 and 1.1 on Android Kitkat. This function is mostly taken from the documentation of
+     * OkHttpClient.Builder.sslSocketFactory(_,_)
+     * <p>
+     * If there is an error, the function will safely fall back to doing nothing and printing the error to the console.
+     *
+     * @param builder The HTTPClient Builder on which TLS is enabled on (will be modified in-place)
+     */
+    private static void enableModernTLS(OkHttpClient.Builder builder) {
+        try {
+            // get the default TrustManager
+            TrustManagerFactory trustManagerFactory = TrustManagerFactory.getInstance(
+                    TrustManagerFactory.getDefaultAlgorithm());
+            trustManagerFactory.init((KeyStore) null);
+            TrustManager[] trustManagers = trustManagerFactory.getTrustManagers();
+            if (trustManagers.length != 1 || !(trustManagers[0] instanceof X509TrustManager)) {
+                throw new IllegalStateException("Unexpected default trust managers:"
+                        + Arrays.toString(trustManagers));
+            }
+            X509TrustManager trustManager = (X509TrustManager) trustManagers[0];
+
+            // insert our own TLSSocketFactory
+            SSLSocketFactory sslSocketFactory = TLSSocketFactoryCompat.getInstance();
+
+            builder.sslSocketFactory(sslSocketFactory, trustManager);
+
+            // This will try to enable all modern CipherSuites(+2 more) that are supported on the device.
+            // Necessary because some servers (e.g. Framatube.org) don't support the old cipher suites.
+            // https://github.com/square/okhttp/issues/4053#issuecomment-402579554
+            List<CipherSuite> cipherSuites = new ArrayList<>();
+            cipherSuites.addAll(ConnectionSpec.MODERN_TLS.cipherSuites());
+            cipherSuites.add(CipherSuite.TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA);
+            cipherSuites.add(CipherSuite.TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA);
+            ConnectionSpec legacyTLS = new ConnectionSpec.Builder(ConnectionSpec.MODERN_TLS)
+                    .cipherSuites(cipherSuites.toArray(new CipherSuite[0]))
+                    .build();
+
+            builder.connectionSpecs(Arrays.asList(legacyTLS, ConnectionSpec.CLEARTEXT));
+        } catch (KeyManagementException | NoSuchAlgorithmException | KeyStoreException e) {
+            if (DEBUG) e.printStackTrace();
+        }
+    }
+}

app/src/main/java/org/schabi/newpipe/ImageDownloader.java

@@ -40,7 +40,7 @@ public class ImageDownloader extends BaseImageDownloader {
     }
 
     protected InputStream getStreamFromNetwork(String imageUri, Object extra) throws IOException {
-        final Downloader downloader = (Downloader) NewPipe.getDownloader();
+        final DownloaderImpl downloader = (DownloaderImpl) NewPipe.getDownloader();
         return downloader.stream(imageUri);
     }
 }

app/src/main/java/org/schabi/newpipe/MainActivity.java

@@ -29,14 +29,17 @@ import android.os.Handler;
 import android.os.Looper;
 import android.preference.PreferenceManager;
 import android.util.Log;
+import android.view.LayoutInflater;
 import android.view.Menu;
-import android.view.MenuInflater;
 import android.view.MenuItem;
 import android.view.View;
 import android.view.Window;
 import android.view.WindowManager;
+import android.widget.AdapterView;
+import android.widget.ArrayAdapter;
 import android.widget.Button;
 import android.widget.ImageView;
+import android.widget.Spinner;
 import android.widget.TextView;
 
 import androidx.annotation.NonNull;
@@ -47,12 +50,14 @@ import androidx.appcompat.widget.Toolbar;
 import androidx.core.view.GravityCompat;
 import androidx.drawerlayout.widget.DrawerLayout;
 import androidx.fragment.app.Fragment;
+import androidx.fragment.app.FragmentManager;
 
 import com.google.android.material.navigation.NavigationView;
 
 import org.schabi.newpipe.extractor.NewPipe;
 import org.schabi.newpipe.extractor.StreamingService;
 import org.schabi.newpipe.extractor.exceptions.ExtractionException;
+import org.schabi.newpipe.extractor.services.peertube.PeertubeInstance;
 import org.schabi.newpipe.fragments.BackPressable;
 import org.schabi.newpipe.fragments.MainFragment;
 import org.schabi.newpipe.fragments.detail.VideoDetailFragment;
@@ -60,12 +65,20 @@ import org.schabi.newpipe.fragments.list.search.SearchFragment;
 import org.schabi.newpipe.report.ErrorActivity;
 import org.schabi.newpipe.util.Constants;
 import org.schabi.newpipe.util.KioskTranslator;
+import org.schabi.newpipe.util.Localization;
 import org.schabi.newpipe.util.NavigationHelper;
+import org.schabi.newpipe.util.PeertubeHelper;
 import org.schabi.newpipe.util.PermissionHelper;
 import org.schabi.newpipe.util.ServiceHelper;
 import org.schabi.newpipe.util.StateSaver;
+import org.schabi.newpipe.util.TLSSocketFactoryCompat;
 import org.schabi.newpipe.util.ThemeHelper;
 
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.schabi.newpipe.util.Localization.assureCorrectAppLanguage;
+
 public class MainActivity extends AppCompatActivity {
     private static final String TAG = "MainActivity";
     public static final boolean DEBUG = !BuildConfig.BUILD_TYPE.equals("release");
@@ -97,8 +110,13 @@ public class MainActivity extends AppCompatActivity {
     protected void onCreate(Bundle savedInstanceState) {
         if (DEBUG) Log.d(TAG, "onCreate() called with: savedInstanceState = [" + savedInstanceState + "]");
 
+        // enable TLS1.1/1.2 for kitkat devices, to fix download and play for mediaCCC sources
+        if (Build.VERSION.SDK_INT == Build.VERSION_CODES.KITKAT) {
+            TLSSocketFactoryCompat.setAsDefault();
+        }
         ThemeHelper.setTheme(this, ServiceHelper.getSelectedServiceId(this));
 
+        assureCorrectAppLanguage(this);
         super.onCreate(savedInstanceState);
         setContentView(R.layout.activity_main);
 
@@ -141,7 +159,7 @@ public class MainActivity extends AppCompatActivity {
                 .add(R.id.menu_tabs_group, ITEM_ID_SUBSCRIPTIONS, ORDER, R.string.tab_subscriptions)
                 .setIcon(ThemeHelper.resolveResourceIdFromAttr(this, R.attr.ic_channel));
         drawerItems.getMenu()
-                .add(R.id.menu_tabs_group, ITEM_ID_FEED, ORDER, R.string.fragment_whats_new)
+                .add(R.id.menu_tabs_group, ITEM_ID_FEED, ORDER, R.string.fragment_feed_title)
                 .setIcon(ThemeHelper.resolveResourceIdFromAttr(this, R.attr.rss));
         drawerItems.getMenu()
                 .add(R.id.menu_tabs_group, ITEM_ID_BOOKMARKS, ORDER, R.string.tab_bookmarks)
@@ -222,7 +240,7 @@ public class MainActivity extends AppCompatActivity {
                 NavigationHelper.openSubscriptionFragment(getSupportFragmentManager());
                 break;
             case ITEM_ID_FEED:
-                NavigationHelper.openWhatsNewFragment(getSupportFragmentManager());
+                NavigationHelper.openFeedFragment(getSupportFragmentManager());
                 break;
             case ITEM_ID_BOOKMARKS:
                 NavigationHelper.openBookmarksFragment(getSupportFragmentManager());
@@ -300,13 +318,57 @@ public class MainActivity extends AppCompatActivity {
             final String title = s.getServiceInfo().getName() +
                     (ServiceHelper.isBeta(s) ? " (beta)" : "");
 
-            drawerItems.getMenu()
+            MenuItem menuItem = drawerItems.getMenu()
                     .add(R.id.menu_services_group, s.getServiceId(), ORDER, title)
                     .setIcon(ServiceHelper.getIcon(s.getServiceId()));
+
+            // peertube specifics
+            if(s.getServiceId() == 3){
+                enhancePeertubeMenu(s, menuItem);
+            }
         }
         drawerItems.getMenu().getItem(ServiceHelper.getSelectedServiceId(this)).setChecked(true);
     }
 
+    private void enhancePeertubeMenu(StreamingService s, MenuItem menuItem) {
+        PeertubeInstance currentInstace = PeertubeHelper.getCurrentInstance();
+        menuItem.setTitle(currentInstace.getName() + (ServiceHelper.isBeta(s) ? " (beta)" : ""));
+        Spinner spinner = (Spinner) LayoutInflater.from(this).inflate(R.layout.instance_spinner_layout, null);
+        List<PeertubeInstance> instances = PeertubeHelper.getInstanceList(this);
+        List<String> items = new ArrayList<>();
+        int defaultSelect = 0;
+        for(PeertubeInstance instance: instances){
+            items.add(instance.getName());
+            if(instance.getUrl().equals(currentInstace.getUrl())){
+                defaultSelect = items.size()-1;
+            }
+        }
+        ArrayAdapter<String> adapter = new ArrayAdapter<>(this, R.layout.instance_spinner_item, items);
+        adapter.setDropDownViewResource(android.R.layout.simple_spinner_dropdown_item);
+        spinner.setAdapter(adapter);
+        spinner.setSelection(defaultSelect, false);
+        spinner.setOnItemSelectedListener(new AdapterView.OnItemSelectedListener() {
+            @Override
+            public void onItemSelected(AdapterView<?> parent, View view, int position, long id) {
+                PeertubeInstance newInstance = instances.get(position);
+                if(newInstance.getUrl().equals(PeertubeHelper.getCurrentInstance().getUrl())) return;
+                PeertubeHelper.selectInstance(newInstance, getApplicationContext());
+                changeService(menuItem);
+                drawer.closeDrawers();
+                new Handler(Looper.getMainLooper()).postDelayed(() -> {
+                    getSupportFragmentManager().popBackStack(null, FragmentManager.POP_BACK_STACK_INCLUSIVE);
+                    recreate();
+                }, 300);
+            }
+
+            @Override
+            public void onNothingSelected(AdapterView<?> parent) {
+
+            }
+        });
+        menuItem.setActionView(spinner);
+    }
+
     private void showTabs() throws ExtractionException {
         serviceArrow.setImageResource(R.drawable.ic_arrow_down_white);
 
@@ -327,7 +389,7 @@ public class MainActivity extends AppCompatActivity {
                 .add(R.id.menu_tabs_group, ITEM_ID_SUBSCRIPTIONS, ORDER, R.string.tab_subscriptions)
                 .setIcon(ThemeHelper.resolveResourceIdFromAttr(this, R.attr.ic_channel));
         drawerItems.getMenu()
-                .add(R.id.menu_tabs_group, ITEM_ID_FEED, ORDER, R.string.fragment_whats_new)
+                .add(R.id.menu_tabs_group, ITEM_ID_FEED, ORDER, R.string.fragment_feed_title)
                 .setIcon(ThemeHelper.resolveResourceIdFromAttr(this, R.attr.rss));
         drawerItems.getMenu()
                 .add(R.id.menu_tabs_group, ITEM_ID_BOOKMARKS, ORDER, R.string.tab_bookmarks)
@@ -358,6 +420,8 @@ public class MainActivity extends AppCompatActivity {
 
     @Override
     protected void onResume() {
+        assureCorrectAppLanguage(this);
+        Localization.init(getApplicationContext()); //change the date format to match the selected language on resume
         super.onResume();
 
         // close drawer on return, and don't show animation, so its looks like the drawer isn't open
@@ -367,6 +431,7 @@ public class MainActivity extends AppCompatActivity {
             String selectedServiceName = NewPipe.getService(
                     ServiceHelper.getSelectedServiceId(this)).getServiceInfo().getName();
             headerServiceView.setText(selectedServiceName);
+            headerServiceView.post(() -> headerServiceView.setSelected(true));
             toggleServiceButton.setContentDescription(
                     getString(R.string.drawer_header_description) + selectedServiceName);
         } catch (Exception e) {
@@ -387,6 +452,10 @@ public class MainActivity extends AppCompatActivity {
             sharedPreferences.edit().putBoolean(Constants.KEY_MAIN_PAGE_CHANGE, false).apply();
             NavigationHelper.openMainActivity(this);
         }
+
+        final boolean isHistoryEnabled = sharedPreferences.getBoolean(
+                getString(R.string.enable_watch_history_key), true);
+        drawerItems.getMenu().findItem(ITEM_ID_HISTORY).setVisible(isHistoryEnabled);
     }
 
     @Override
@@ -489,8 +558,6 @@ public class MainActivity extends AppCompatActivity {
         if (!(fragment instanceof SearchFragment)) {
             findViewById(R.id.toolbar).findViewById(R.id.toolbar_search_container).setVisibility(View.GONE);
 
-            MenuInflater inflater = getMenuInflater();
-            inflater.inflate(R.menu.main_menu, menu);
         }
 
         ActionBar actionBar = getSupportActionBar();
@@ -512,14 +579,6 @@ public class MainActivity extends AppCompatActivity {
             case android.R.id.home:
                 onHomeButtonPressed();
                 return true;
-            case R.id.action_show_downloads:
-                    return NavigationHelper.openDownloads(this);
-            case R.id.action_history:
-                    NavigationHelper.openStatisticFragment(getSupportFragmentManager());
-                    return true;
-            case R.id.action_settings:
-                    NavigationHelper.openSettings(this);
-                    return true;
             default:
                 return super.onOptionsItemSelected(item);
         }

app/src/main/java/org/schabi/newpipe/NewPipeDatabase.java

@@ -1,13 +1,16 @@
 package org.schabi.newpipe;
 
-import androidx.room.Room;
 import android.content.Context;
+import android.database.Cursor;
+
 import androidx.annotation.NonNull;
+import androidx.room.Room;
 
 import org.schabi.newpipe.database.AppDatabase;
 
 import static org.schabi.newpipe.database.AppDatabase.DATABASE_NAME;
-import static org.schabi.newpipe.database.Migrations.MIGRATION_11_12;
+import static org.schabi.newpipe.database.Migrations.MIGRATION_1_2;
+import static org.schabi.newpipe.database.Migrations.MIGRATION_2_3;
 
 public final class NewPipeDatabase {
 
@@ -20,8 +23,7 @@ public final class NewPipeDatabase {
     private static AppDatabase getDatabase(Context context) {
         return Room
                 .databaseBuilder(context.getApplicationContext(), AppDatabase.class, DATABASE_NAME)
-                .addMigrations(MIGRATION_11_12)
-                .fallbackToDestructiveMigration()
+                .addMigrations(MIGRATION_1_2, MIGRATION_2_3)
                 .build();
     }
 
@@ -39,4 +41,14 @@ public final class NewPipeDatabase {
 
         return result;
     }
+
+    public static void checkpoint() {
+        if (databaseInstance == null) {
+            throw new IllegalStateException("database is not initialized");
+        }
+        Cursor c = databaseInstance.query("pragma wal_checkpoint(full)", null);
+        if (c.moveToFirst() && c.getInt(0) == 1) {
+            throw new RuntimeException("Checkpoint was blocked from completing");
+        }
+    }
 }

app/src/main/java/org/schabi/newpipe/ReCaptchaActivity.java

@@ -1,20 +1,25 @@
 package org.schabi.newpipe;
 
-import android.app.Activity;
 import android.content.Intent;
-import android.graphics.Bitmap;
 import android.os.Build;
 import android.os.Bundle;
 import androidx.core.app.NavUtils;
 import androidx.appcompat.app.ActionBar;
 import androidx.appcompat.app.AppCompatActivity;
 import androidx.appcompat.widget.Toolbar;
+
+import android.util.Log;
+import android.view.Menu;
 import android.view.MenuItem;
 import android.webkit.CookieManager;
 import android.webkit.WebSettings;
 import android.webkit.WebView;
 import android.webkit.WebViewClient;
 
+import org.schabi.newpipe.util.ThemeHelper;
+
+import androidx.annotation.NonNull;
+
 /*
  * Created by beneth <bmauduit@beneth.fr> on 06.12.16.
  *
@@ -37,48 +42,46 @@ import android.webkit.WebViewClient;
 public class ReCaptchaActivity extends AppCompatActivity {
     public static final int RECAPTCHA_REQUEST = 10;
     public static final String RECAPTCHA_URL_EXTRA = "recaptcha_url_extra";
-
     public static final String TAG = ReCaptchaActivity.class.toString();
     public static final String YT_URL = "https://www.youtube.com";
 
-    private String url;
+    private WebView webView;
+    private String foundCookies = "";
 
     @Override
     protected void onCreate(Bundle savedInstanceState) {
+        ThemeHelper.setTheme(this);
         super.onCreate(savedInstanceState);
         setContentView(R.layout.activity_recaptcha);
+        Toolbar toolbar = findViewById(R.id.toolbar);
+        setSupportActionBar(toolbar);
 
-        url = getIntent().getStringExtra(RECAPTCHA_URL_EXTRA);
+        String url = getIntent().getStringExtra(RECAPTCHA_URL_EXTRA);
         if (url == null || url.isEmpty()) {
             url = YT_URL;
         }
 
-
-        // Set return to Cancel by default
+        // set return to Cancel by default
         setResult(RESULT_CANCELED);
 
-        Toolbar toolbar = findViewById(R.id.toolbar);
-        setSupportActionBar(toolbar);
 
-        ActionBar actionBar = getSupportActionBar();
-        if (actionBar != null) {
-            actionBar.setDisplayHomeAsUpEnabled(true);
-            actionBar.setTitle(R.string.reCaptcha_title);
-            actionBar.setDisplayShowTitleEnabled(true);
-        }
+        webView = findViewById(R.id.reCaptchaWebView);
 
-        WebView myWebView = findViewById(R.id.reCaptchaWebView);
-
-        // Enable Javascript
-        WebSettings webSettings = myWebView.getSettings();
+        // enable Javascript
+        WebSettings webSettings = webView.getSettings();
         webSettings.setJavaScriptEnabled(true);
 
-        ReCaptchaWebViewClient webClient = new ReCaptchaWebViewClient(this);
-        myWebView.setWebViewClient(webClient);
+        webView.setWebViewClient(new WebViewClient() {
+            @Override
+            public void onPageFinished(WebView view, String url) {
+                super.onPageFinished(view, url);
+                handleCookies(url);
+            }
+        });
 
-        // Cleaning cache, history and cookies from webView
-        myWebView.clearCache(true);
-        myWebView.clearHistory();
+        // cleaning cache, history and cookies from webView
+        webView.clearCache(true);
+        webView.clearHistory();
         android.webkit.CookieManager cookieManager = CookieManager.getInstance();
         if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.LOLLIPOP) {
             cookieManager.removeAllCookies(aBoolean -> {});
@@ -86,77 +89,82 @@ public class ReCaptchaActivity extends AppCompatActivity {
             cookieManager.removeAllCookie();
         }
 
-        myWebView.loadUrl(url);
+        webView.loadUrl(url);
     }
 
-    private class ReCaptchaWebViewClient extends WebViewClient {
-        private final Activity context;
-        private String mCookies;
+    @Override
+    public boolean onCreateOptionsMenu(Menu menu) {
+        getMenuInflater().inflate(R.menu.menu_recaptcha, menu);
 
-        ReCaptchaWebViewClient(Activity ctx) {
-            context = ctx;
+        ActionBar actionBar = getSupportActionBar();
+        if (actionBar != null) {
+            actionBar.setDisplayHomeAsUpEnabled(false);
+            actionBar.setTitle(R.string.title_activity_recaptcha);
+            actionBar.setSubtitle(R.string.subtitle_activity_recaptcha);
         }
 
-        @Override
-        public void onPageStarted(WebView view, String url, Bitmap favicon) {
-            // TODO: Start Loader
-            super.onPageStarted(view, url, favicon);
-        }
+        return true;
+    }
 
-        @Override
-        public void onPageFinished(WebView view, String url) {
-            String cookies = CookieManager.getInstance().getCookie(url);
-
-            // TODO: Stop Loader
-
-            // find cookies : s_gl & goojf and Add cookies to Downloader
-            if (find_access_cookies(cookies)) {
-                // Give cookies to Downloader class
-                Downloader.getInstance().setCookies(mCookies);
-
-                // Closing activity and return to parent
-                setResult(RESULT_OK);
-                finish();
-            }
-        }
-
-        private boolean find_access_cookies(String cookies) {
-            boolean ret = false;
-            String c_s_gl = "";
-            String c_goojf = "";
-
-            String[] parts = cookies.split("; ");
-            for (String part : parts) {
-                if (part.trim().startsWith("s_gl")) {
-                    c_s_gl = part.trim();
-                }
-                if (part.trim().startsWith("goojf")) {
-                    c_goojf = part.trim();
-                }
-            }
-            if (c_s_gl.length() > 0 && c_goojf.length() > 0) {
-                ret = true;
-                //mCookies = c_s_gl + "; " + c_goojf;
-                // Youtube seems to also need the other cookies:
-                mCookies = cookies;
-            }
-
-            return ret;
-        }
+    @Override
+    public void onBackPressed() {
+        saveCookiesAndFinish();
     }
 
     @Override
     public boolean onOptionsItemSelected(MenuItem item) {
         int id = item.getItemId();
         switch (id) {
-            case android.R.id.home: {
-                Intent intent = new Intent(this, org.schabi.newpipe.MainActivity.class);
-                intent.addFlags(Intent.FLAG_ACTIVITY_CLEAR_TOP);
-                NavUtils.navigateUpTo(this, intent);
+            case R.id.menu_item_done:
+                saveCookiesAndFinish();
                 return true;
-            }
             default:
                 return false;
         }
     }
+
+    private void saveCookiesAndFinish() {
+        handleCookies(webView.getUrl()); // try to get cookies of unclosed page
+        if (!foundCookies.isEmpty()) {
+            // give cookies to Downloader class
+            DownloaderImpl.getInstance().setCookies(foundCookies);
+            setResult(RESULT_OK);
+        }
+
+        Intent intent = new Intent(this, org.schabi.newpipe.MainActivity.class);
+        intent.addFlags(Intent.FLAG_ACTIVITY_CLEAR_TOP);
+        NavUtils.navigateUpTo(this, intent);
+    }
+
+
+
+    private void handleCookies(String url) {
+        String cookies = CookieManager.getInstance().getCookie(url);
+        if (MainActivity.DEBUG) Log.d(TAG, "handleCookies: url=" + url + "; cookies=" + (cookies == null ? "null" : cookies));
+        if (cookies == null) return;
+
+        addYoutubeCookies(cookies);
+        // add other methods to extract cookies here
+    }
+
+    private void addYoutubeCookies(@NonNull String cookies) {
+        if (cookies.contains("s_gl=") || cookies.contains("goojf=") || cookies.contains("VISITOR_INFO1_LIVE=")) {
+            // youtube seems to also need the other cookies:
+            addCookie(cookies);
+        }
+    }
+
+    private void addCookie(String cookie) {
+        if (foundCookies.contains(cookie)) {
+            return;
+        }
+
+        if (foundCookies.isEmpty() || foundCookies.endsWith("; ")) {
+            foundCookies += cookie;
+        } else if (foundCookies.endsWith(";")) {
+            foundCookies += " " + cookie;
+        } else {
+            foundCookies += "; " + cookie;
+        }
+    }
 }

app/src/main/java/org/schabi/newpipe/RouterActivity.java

@@ -9,12 +9,6 @@ import android.content.SharedPreferences;
 import android.content.pm.PackageManager;
 import android.os.Bundle;
 import android.preference.PreferenceManager;
-import androidx.annotation.DrawableRes;
-import androidx.annotation.NonNull;
-import androidx.annotation.Nullable;
-import androidx.core.app.NotificationCompat;
-import androidx.appcompat.app.AlertDialog;
-import androidx.appcompat.app.AppCompatActivity;
 import android.text.TextUtils;
 import android.view.ContextThemeWrapper;
 import android.view.LayoutInflater;
@@ -26,6 +20,12 @@ import android.widget.RadioButton;
 import android.widget.RadioGroup;
 import android.widget.Toast;
 
+import androidx.annotation.DrawableRes;
+import androidx.annotation.NonNull;
+import androidx.annotation.Nullable;
+import androidx.appcompat.app.AlertDialog;
+import androidx.appcompat.app.AppCompatActivity;
+import androidx.core.app.NotificationCompat;
 import androidx.fragment.app.FragmentManager;
 
 import org.schabi.newpipe.download.DownloadDialog;
@@ -49,12 +49,11 @@ import org.schabi.newpipe.util.ListHelper;
 import org.schabi.newpipe.util.NavigationHelper;
 import org.schabi.newpipe.util.PermissionHelper;
 import org.schabi.newpipe.util.ThemeHelper;
+import org.schabi.newpipe.util.urlfinder.UrlFinder;
 
 import java.io.Serializable;
 import java.util.ArrayList;
 import java.util.Arrays;
-import java.util.Collection;
-import java.util.HashSet;
 import java.util.List;
 
 import icepick.Icepick;
@@ -625,78 +624,18 @@ public class RouterActivity extends AppCompatActivity {
     // Utils
     //////////////////////////////////////////////////////////////////////////*/
 
-    /**
-     * Removes invisible separators (\p{Z}) and punctuation characters including
-     * brackets (\p{P}). See http://www.regular-expressions.info/unicode.html for
-     * more details.
-     */
-    private final static String REGEX_REMOVE_FROM_URL = "[\\p{Z}\\p{P}]";
-
+    @Nullable
     private String getUrl(Intent intent) {
-        // first gather data and find service
-        String videoUrl = null;
+        String foundUrl = null;
         if (intent.getData() != null) {
-            // this means the video was called though another app
-            videoUrl = intent.getData().toString();
+            // Called from another app
+            foundUrl = intent.getData().toString();
         } else if (intent.getStringExtra(Intent.EXTRA_TEXT) != null) {
-            //this means that vidoe was called through share menu
-            String extraText = intent.getStringExtra(Intent.EXTRA_TEXT);
-            final String[] uris = getUris(extraText);
-            videoUrl = uris.length > 0 ? uris[0] : null;
+            // Called from the share menu
+            final String extraText = intent.getStringExtra(Intent.EXTRA_TEXT);
+            foundUrl = UrlFinder.firstUrlFromInput(extraText);
         }
 
-        return videoUrl;
-    }
-
-    private String removeHeadingGibberish(final String input) {
-        int start = 0;
-        for (int i = input.indexOf("://") - 1; i >= 0; i--) {
-            if (!input.substring(i, i + 1).matches("\\p{L}")) {
-                start = i + 1;
-                break;
-            }
-        }
-        return input.substring(start, input.length());
-    }
-
-    private String trim(final String input) {
-        if (input == null || input.length() < 1) {
-            return input;
-        } else {
-            String output = input;
-            while (output.length() > 0 && output.substring(0, 1).matches(REGEX_REMOVE_FROM_URL)) {
-                output = output.substring(1);
-            }
-            while (output.length() > 0
-                    && output.substring(output.length() - 1, output.length()).matches(REGEX_REMOVE_FROM_URL)) {
-                output = output.substring(0, output.length() - 1);
-            }
-            return output;
-        }
-    }
-
-    /**
-     * Retrieves all Strings which look remotely like URLs from a text.
-     * Used if NewPipe was called through share menu.
-     *
-     * @param sharedText text to scan for URLs.
-     * @return potential URLs
-     */
-    protected String[] getUris(final String sharedText) {
-        final Collection<String> result = new HashSet<>();
-        if (sharedText != null) {
-            final String[] array = sharedText.split("\\p{Space}");
-            for (String s : array) {
-                s = trim(s);
-                if (s.length() != 0) {
-                    if (s.matches(".+://.+")) {
-                        result.add(removeHeadingGibberish(s));
-                    } else if (s.matches(".+\\..+")) {
-                        result.add("http://" + s);
-                    }
-                }
-            }
-        }
-        return result.toArray(new String[result.size()]);
+        return foundUrl;
     }
 }

app/src/main/java/org/schabi/newpipe/about/AboutActivity.java

@@ -22,27 +22,31 @@ import android.widget.TextView;
 
 import org.schabi.newpipe.BuildConfig;
 import org.schabi.newpipe.R;
-import org.schabi.newpipe.util.NavigationHelper;
 import org.schabi.newpipe.util.ThemeHelper;
 
+import static org.schabi.newpipe.util.Localization.assureCorrectAppLanguage;
+
 public class AboutActivity extends AppCompatActivity {
 
     /**
      * List of all software components
      */
     private static final SoftwareComponent[] SOFTWARE_COMPONENTS = new SoftwareComponent[]{
-            new SoftwareComponent("Giga Get", "2014", "Peter Cai", "https://github.com/PaperAirplane-Dev-Team/GigaGet", StandardLicenses.GPL2),
-            new SoftwareComponent("NewPipe Extractor", "2017", "Christian Schabesberger", "https://github.com/TeamNewPipe/NewPipeExtractor", StandardLicenses.GPL3),
+            new SoftwareComponent("Giga Get", "2014 - 2015", "Peter Cai", "https://github.com/PaperAirplane-Dev-Team/GigaGet", StandardLicenses.GPL2),
+            new SoftwareComponent("NewPipe Extractor", "2017 - 2020", "Christian Schabesberger", "https://github.com/TeamNewPipe/NewPipeExtractor", StandardLicenses.GPL3),
             new SoftwareComponent("Jsoup", "2017", "Jonathan Hedley", "https://github.com/jhy/jsoup", StandardLicenses.MIT),
             new SoftwareComponent("Rhino", "2015", "Mozilla", "https://www.mozilla.org/rhino/", StandardLicenses.MPL2),
             new SoftwareComponent("ACRA", "2013", "Kevin Gaudin", "http://www.acra.ch", StandardLicenses.APACHE2),
             new SoftwareComponent("Universal Image Loader", "2011 - 2015", "Sergey Tarasevich", "https://github.com/nostra13/Android-Universal-Image-Loader", StandardLicenses.APACHE2),
-            new SoftwareComponent("CircleImageView", "2014 - 2017", "Henning Dodenhof", "https://github.com/hdodenhof/CircleImageView", StandardLicenses.APACHE2),
+            new SoftwareComponent("CircleImageView", "2014 - 2020", "Henning Dodenhof", "https://github.com/hdodenhof/CircleImageView", StandardLicenses.APACHE2),
             new SoftwareComponent("NoNonsense-FilePicker", "2016", "Jonas Kalderstam", "https://github.com/spacecowboy/NoNonsense-FilePicker", StandardLicenses.MPL2),
-            new SoftwareComponent("ExoPlayer", "2014-2017", "Google Inc", "https://github.com/google/ExoPlayer", StandardLicenses.APACHE2),
-            new SoftwareComponent("RxAndroid", "2015", "The RxAndroid authors", "https://github.com/ReactiveX/RxAndroid", StandardLicenses.APACHE2),
-            new SoftwareComponent("RxJava", "2016-present", "RxJava Contributors", "https://github.com/ReactiveX/RxJava", StandardLicenses.APACHE2),
-            new SoftwareComponent("RxBinding", "2015", "Jake Wharton", "https://github.com/JakeWharton/RxBinding", StandardLicenses.APACHE2)
+            new SoftwareComponent("ExoPlayer", "2014 - 2020", "Google Inc", "https://github.com/google/ExoPlayer", StandardLicenses.APACHE2),
+            new SoftwareComponent("RxAndroid", "2015 - 2018", "The RxAndroid authors", "https://github.com/ReactiveX/RxAndroid", StandardLicenses.APACHE2),
+            new SoftwareComponent("RxJava", "2016 - 2020", "RxJava Contributors", "https://github.com/ReactiveX/RxJava", StandardLicenses.APACHE2),
+            new SoftwareComponent("RxBinding", "2015 - 2018", "Jake Wharton", "https://github.com/JakeWharton/RxBinding", StandardLicenses.APACHE2),
+            new SoftwareComponent("PrettyTime", "2012 - 2020", "Lincoln Baxter, III", "https://github.com/ocpsoft/prettytime", StandardLicenses.APACHE2),
+            new SoftwareComponent("Markwon", "2017 - 2020", "Noties", "https://github.com/noties/Markwon", StandardLicenses.APACHE2),
+            new SoftwareComponent("Groupie", "2016", "Lisa Wray", "https://github.com/lisawray/groupie", StandardLicenses.MIT)
     };
 
     /**
@@ -62,8 +66,10 @@ public class AboutActivity extends AppCompatActivity {
 
     @Override
     protected void onCreate(Bundle savedInstanceState) {
+        assureCorrectAppLanguage(this);
         super.onCreate(savedInstanceState);
         ThemeHelper.setTheme(this);
+        this.setTitle(getString(R.string.title_activity_about));
 
         setContentView(R.layout.activity_about);
 
@@ -83,13 +89,6 @@ public class AboutActivity extends AppCompatActivity {
     }
 
 
-    @Override
-    public boolean onCreateOptionsMenu(Menu menu) {
-        // Inflate the menu; this adds items to the action bar if it is present.
-        getMenuInflater().inflate(R.menu.menu_about, menu);
-        return true;
-    }
-
     @Override
     public boolean onOptionsItemSelected(MenuItem item) {
 
@@ -99,11 +98,6 @@ public class AboutActivity extends AppCompatActivity {
             case android.R.id.home:
                 finish();
                 return true;
-            case R.id.action_settings:
-                NavigationHelper.openSettings(this);
-                return true;
-            case R.id.action_show_downloads:
-                return NavigationHelper.openDownloads(this);
         }
 
         return super.onOptionsItemSelected(item);

app/src/main/java/org/schabi/newpipe/about/LicenseFragmentHelper.java

@@ -3,6 +3,7 @@ package org.schabi.newpipe.about;
 import android.app.Activity;
 import android.content.Context;
 import android.content.DialogInterface;
+import android.content.res.Resources;
 import android.os.AsyncTask;
 import androidx.annotation.Nullable;
 import androidx.appcompat.app.AlertDialog;
@@ -14,6 +15,8 @@ import java.io.BufferedReader;
 import java.io.InputStreamReader;
 import java.lang.ref.WeakReference;
 
+import static org.schabi.newpipe.util.Localization.assureCorrectAppLanguage;
+
 public class LicenseFragmentHelper extends AsyncTask<Object, Void, Integer> {
 
     final WeakReference<Activity> weakReference;
@@ -55,15 +58,15 @@ public class LicenseFragmentHelper extends AsyncTask<Object, Void, Integer> {
         wv.loadData(webViewData, "text/html; charset=UTF-8", null);
 
         alert.setView(wv);
-        alert.setNegativeButton(android.R.string.ok, new DialogInterface.OnClickListener() {
-            @Override
-            public void onClick(DialogInterface dialog, int which) {
-                dialog.dismiss();
-            }
-        });
+        assureCorrectAppLanguage(activity.getApplicationContext());
+        alert.setNegativeButton(getFinishString(activity), (dialog, which) -> dialog.dismiss());
         alert.show();
     }
 
+    private static String getFinishString(Activity activity) {
+        return activity.getApplicationContext().getResources().getString(R.string.finish);
+    }
+
     /**
      * @param context the context to use
      * @param license the license

app/src/main/java/org/schabi/newpipe/database/AppDatabase.java

@@ -4,6 +4,12 @@ import androidx.room.Database;
 import androidx.room.RoomDatabase;
 import androidx.room.TypeConverters;
 
+import org.schabi.newpipe.database.feed.dao.FeedDAO;
+import org.schabi.newpipe.database.feed.dao.FeedGroupDAO;
+import org.schabi.newpipe.database.feed.model.FeedEntity;
+import org.schabi.newpipe.database.feed.model.FeedGroupEntity;
+import org.schabi.newpipe.database.feed.model.FeedGroupSubscriptionEntity;
+import org.schabi.newpipe.database.feed.model.FeedLastUpdatedEntity;
 import org.schabi.newpipe.database.history.dao.SearchHistoryDAO;
 import org.schabi.newpipe.database.history.dao.StreamHistoryDAO;
 import org.schabi.newpipe.database.history.model.SearchHistoryEntry;
@@ -21,35 +27,33 @@ import org.schabi.newpipe.database.stream.model.StreamStateEntity;
 import org.schabi.newpipe.database.subscription.SubscriptionDAO;
 import org.schabi.newpipe.database.subscription.SubscriptionEntity;
 
-import static org.schabi.newpipe.database.Migrations.DB_VER_12_0;
+import static org.schabi.newpipe.database.Migrations.DB_VER_3;
 
 @TypeConverters({Converters.class})
 @Database(
         entities = {
                 SubscriptionEntity.class, SearchHistoryEntry.class,
                 StreamEntity.class, StreamHistoryEntity.class, StreamStateEntity.class,
-                PlaylistEntity.class, PlaylistStreamEntity.class, PlaylistRemoteEntity.class
+                PlaylistEntity.class, PlaylistStreamEntity.class, PlaylistRemoteEntity.class,
+                FeedEntity.class, FeedGroupEntity.class, FeedGroupSubscriptionEntity.class,
+                FeedLastUpdatedEntity.class
         },
-        version = DB_VER_12_0,
-        exportSchema = false
+        version = DB_VER_3
 )
 public abstract class AppDatabase extends RoomDatabase {
-
     public static final String DATABASE_NAME = "newpipe.db";
 
-    public abstract SubscriptionDAO subscriptionDAO();
-
     public abstract SearchHistoryDAO searchHistoryDAO();
 
     public abstract StreamDAO streamDAO();
-
     public abstract StreamHistoryDAO streamHistoryDAO();
-
     public abstract StreamStateDAO streamStateDAO();
 
     public abstract PlaylistDAO playlistDAO();
-
     public abstract PlaylistStreamDAO playlistStreamDAO();
-
     public abstract PlaylistRemoteDAO playlistRemoteDAO();
+
+    public abstract FeedDAO feedDAO();
+    public abstract FeedGroupDAO feedGroupDAO();
+    public abstract SubscriptionDAO subscriptionDAO();
 }

app/src/main/java/org/schabi/newpipe/database/Converters.java

@@ -3,6 +3,7 @@ package org.schabi.newpipe.database;
 import androidx.room.TypeConverter;
 
 import org.schabi.newpipe.extractor.stream.StreamType;
+import org.schabi.newpipe.local.subscription.FeedGroupIcon;
 
 import java.util.Date;
 
@@ -37,4 +38,18 @@ public class Converters {
     public static String stringOf(StreamType streamType) {
         return streamType.name();
     }
+
+    @TypeConverter
+    public static Integer integerOf(FeedGroupIcon feedGroupIcon) {
+        return feedGroupIcon.getId();
+    }
+
+    @TypeConverter
+    public static FeedGroupIcon feedGroupIconOf(Integer id) {
+        for (FeedGroupIcon icon : FeedGroupIcon.values()) {
+            if (icon.getId() == id) return icon;
+        }
+
+        throw new IllegalArgumentException("There's no feed group icon with the id \"" + id + "\"");
+    }
 }

app/src/main/java/org/schabi/newpipe/database/Migrations.java

@@ -8,14 +8,14 @@ import android.util.Log;
 import org.schabi.newpipe.BuildConfig;
 
 public class Migrations {
-
-    public static final int DB_VER_11_0 = 1;
-    public static final int DB_VER_12_0 = 2;
+    public static final int DB_VER_1 = 1;
+    public static final int DB_VER_2 = 2;
+    public static final int DB_VER_3 = 3;
 
     public static final boolean DEBUG = !BuildConfig.BUILD_TYPE.equals("release");
     private static final String TAG = Migrations.class.getName();
 
-    public static final Migration MIGRATION_11_12 = new Migration(DB_VER_11_0, DB_VER_12_0) {
+    public static final Migration MIGRATION_1_2 = new Migration(DB_VER_1, DB_VER_2) {
         @Override
         public void migrate(@NonNull SupportSQLiteDatabase database) {
             if(DEBUG) {
@@ -71,4 +71,32 @@ public class Migrations {
             }
         }
     };
+
+    public static final Migration MIGRATION_2_3 = new Migration(DB_VER_2, DB_VER_3) {
+        @Override
+        public void migrate(@NonNull SupportSQLiteDatabase database) {
+            // Add NOT NULLs and new fields
+            database.execSQL("CREATE TABLE IF NOT EXISTS streams_new " +
+                    "(uid INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL, service_id INTEGER NOT NULL, url TEXT NOT NULL, title TEXT NOT NULL, stream_type TEXT NOT NULL," +
+                    " duration INTEGER NOT NULL, uploader TEXT NOT NULL, thumbnail_url TEXT, view_count INTEGER, textual_upload_date TEXT, upload_date INTEGER," +
+                    " is_upload_date_approximation INTEGER)");
+
+            database.execSQL("INSERT INTO streams_new (uid, service_id, url, title, stream_type, duration, uploader, thumbnail_url, view_count, textual_upload_date, upload_date, is_upload_date_approximation)"+
+                    " SELECT uid, service_id, url, title, stream_type, duration, uploader, thumbnail_url, NULL, NULL, NULL, NULL FROM streams");
+
+            database.execSQL("DROP TABLE streams");
+            database.execSQL("ALTER TABLE streams_new RENAME TO streams");
+            database.execSQL("CREATE UNIQUE INDEX index_streams_service_id_url ON streams (service_id, url)");
+
+            // Tables for feed feature
+            database.execSQL("CREATE TABLE IF NOT EXISTS feed (stream_id INTEGER NOT NULL, subscription_id INTEGER NOT NULL, PRIMARY KEY(stream_id, subscription_id), FOREIGN KEY(stream_id) REFERENCES streams(uid) ON UPDATE CASCADE ON DELETE CASCADE DEFERRABLE INITIALLY DEFERRED, FOREIGN KEY(subscription_id) REFERENCES subscriptions(uid) ON UPDATE CASCADE ON DELETE CASCADE DEFERRABLE INITIALLY DEFERRED)");
+            database.execSQL("CREATE INDEX index_feed_subscription_id ON feed (subscription_id)");
+            database.execSQL("CREATE TABLE IF NOT EXISTS feed_group (uid INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL, name TEXT NOT NULL, icon_id INTEGER NOT NULL, sort_order INTEGER NOT NULL)");
+            database.execSQL("CREATE INDEX index_feed_group_sort_order ON feed_group (sort_order)");
+            database.execSQL("CREATE TABLE IF NOT EXISTS feed_group_subscription_join (group_id INTEGER NOT NULL, subscription_id INTEGER NOT NULL, PRIMARY KEY(group_id, subscription_id), FOREIGN KEY(group_id) REFERENCES feed_group(uid) ON UPDATE CASCADE ON DELETE CASCADE DEFERRABLE INITIALLY DEFERRED, FOREIGN KEY(subscription_id) REFERENCES subscriptions(uid) ON UPDATE CASCADE ON DELETE CASCADE DEFERRABLE INITIALLY DEFERRED)");
+            database.execSQL("CREATE INDEX index_feed_group_subscription_join_subscription_id ON feed_group_subscription_join (subscription_id)");
+            database.execSQL("CREATE TABLE IF NOT EXISTS feed_last_updated (subscription_id INTEGER NOT NULL, last_updated INTEGER, PRIMARY KEY(subscription_id), FOREIGN KEY(subscription_id) REFERENCES subscriptions(uid) ON UPDATE CASCADE ON DELETE CASCADE DEFERRABLE INITIALLY DEFERRED)");
+        }
+    };
+
 }

app/src/main/java/org/schabi/newpipe/database/feed/dao/FeedDAO.kt

@@ -0,0 +1,147 @@
+package org.schabi.newpipe.database.feed.dao
+
+import androidx.room.*
+import io.reactivex.Flowable
+import org.schabi.newpipe.database.feed.model.FeedEntity
+import org.schabi.newpipe.database.feed.model.FeedLastUpdatedEntity
+import org.schabi.newpipe.database.stream.model.StreamEntity
+import org.schabi.newpipe.database.subscription.SubscriptionEntity
+import java.util.*
+
+@Dao
+abstract class FeedDAO {
+    @Query("DELETE FROM feed")
+    abstract fun deleteAll(): Int
+
+    @Query("""
+        SELECT s.* FROM streams s
+
+        INNER JOIN feed f
+        ON s.uid = f.stream_id
+
+        ORDER BY s.upload_date IS NULL DESC, s.upload_date DESC, s.uploader ASC
+
+        LIMIT 500
+        """)
+    abstract fun getAllStreams(): Flowable<List<StreamEntity>>
+
+    @Query("""
+        SELECT s.* FROM streams s
+
+        INNER JOIN feed f
+        ON s.uid = f.stream_id
+
+        INNER JOIN feed_group_subscription_join fgs
+        ON fgs.subscription_id = f.subscription_id
+
+        INNER JOIN feed_group fg
+        ON fg.uid = fgs.group_id
+
+        WHERE fgs.group_id = :groupId
+
+        ORDER BY s.upload_date IS NULL DESC, s.upload_date DESC, s.uploader ASC
+        LIMIT 500
+        """)
+    abstract fun getAllStreamsFromGroup(groupId: Long): Flowable<List<StreamEntity>>
+
+    @Query("""
+        DELETE FROM feed WHERE
+
+        feed.stream_id IN (
+            SELECT s.uid FROM streams s
+
+            INNER JOIN feed f
+            ON s.uid = f.stream_id
+
+            WHERE s.upload_date < :date
+        )
+        """)
+    abstract fun unlinkStreamsOlderThan(date: Date)
+
+    @Query("""
+        DELETE FROM feed
+        
+        WHERE feed.subscription_id = :subscriptionId
+
+        AND feed.stream_id IN (
+            SELECT s.uid FROM streams s
+
+            INNER JOIN feed f
+            ON s.uid = f.stream_id
+
+            WHERE s.stream_type = "LIVE_STREAM" OR s.stream_type = "AUDIO_LIVE_STREAM"
+        )
+        """)
+    abstract fun unlinkOldLivestreams(subscriptionId: Long)
+
+    @Insert(onConflict = OnConflictStrategy.IGNORE)
+    abstract fun insert(feedEntity: FeedEntity)
+
+    @Insert(onConflict = OnConflictStrategy.IGNORE)
+    abstract fun insertAll(entities: List<FeedEntity>): List<Long>
+
+    @Insert(onConflict = OnConflictStrategy.IGNORE)
+    internal abstract fun insertLastUpdated(lastUpdatedEntity: FeedLastUpdatedEntity): Long
+
+    @Update(onConflict = OnConflictStrategy.IGNORE)
+    internal abstract fun updateLastUpdated(lastUpdatedEntity: FeedLastUpdatedEntity)
+
+    @Transaction
+    open fun setLastUpdatedForSubscription(lastUpdatedEntity: FeedLastUpdatedEntity) {
+        val id = insertLastUpdated(lastUpdatedEntity)
+
+        if (id == -1L) {
+            updateLastUpdated(lastUpdatedEntity)
+        }
+    }
+
+    @Query("""
+        SELECT MIN(lu.last_updated) FROM feed_last_updated lu
+
+        INNER JOIN feed_group_subscription_join fgs
+        ON fgs.subscription_id = lu.subscription_id AND fgs.group_id = :groupId
+        """)
+    abstract fun oldestSubscriptionUpdate(groupId: Long): Flowable<List<Date>>
+
+    @Query("SELECT MIN(last_updated) FROM feed_last_updated")
+    abstract fun oldestSubscriptionUpdateFromAll(): Flowable<List<Date>>
+
+    @Query("SELECT COUNT(*) FROM feed_last_updated WHERE last_updated IS NULL")
+    abstract fun notLoadedCount(): Flowable<Long>
+
+    @Query("""
+        SELECT COUNT(*) FROM subscriptions s
+        
+        INNER JOIN feed_group_subscription_join fgs
+        ON s.uid = fgs.subscription_id AND fgs.group_id = :groupId
+
+        LEFT JOIN feed_last_updated lu
+        ON s.uid = lu.subscription_id 
+
+        WHERE lu.last_updated IS NULL
+        """)
+    abstract fun notLoadedCountForGroup(groupId: Long): Flowable<Long>
+
+    @Query("""
+        SELECT s.* FROM subscriptions s
+
+        LEFT JOIN feed_last_updated lu
+        ON s.uid = lu.subscription_id 
+
+        WHERE lu.last_updated IS NULL OR lu.last_updated < :outdatedThreshold
+        """)
+    abstract fun getAllOutdated(outdatedThreshold: Date): Flowable<List<SubscriptionEntity>>
+
+    @Query("""
+        SELECT s.* FROM subscriptions s
+
+        INNER JOIN feed_group_subscription_join fgs
+        ON s.uid = fgs.subscription_id AND fgs.group_id = :groupId
+
+        LEFT JOIN feed_last_updated lu
+        ON s.uid = lu.subscription_id
+
+        WHERE lu.last_updated IS NULL OR lu.last_updated < :outdatedThreshold
+        """)
+    abstract fun getAllOutdatedForGroup(groupId: Long, outdatedThreshold: Date): Flowable<List<SubscriptionEntity>>
+}

app/src/main/java/org/schabi/newpipe/database/feed/dao/FeedGroupDAO.kt

@@ -0,0 +1,62 @@
+package org.schabi.newpipe.database.feed.dao
+
+import androidx.room.*
+import io.reactivex.Flowable
+import io.reactivex.Maybe
+import org.schabi.newpipe.database.feed.model.FeedGroupEntity
+import org.schabi.newpipe.database.feed.model.FeedGroupSubscriptionEntity
+
+@Dao
+abstract class FeedGroupDAO {
+
+    @Query("SELECT * FROM feed_group ORDER BY sort_order ASC")
+    abstract fun getAll(): Flowable<List<FeedGroupEntity>>
+
+    @Query("SELECT * FROM feed_group WHERE uid = :groupId")
+    abstract fun getGroup(groupId: Long): Maybe<FeedGroupEntity>
+
+    @Transaction
+    open fun insert(feedGroupEntity: FeedGroupEntity): Long {
+        val nextSortOrder = nextSortOrder()
+        feedGroupEntity.sortOrder = nextSortOrder
+        return insertInternal(feedGroupEntity)
+    }
+
+    @Update(onConflict = OnConflictStrategy.IGNORE)
+    abstract fun update(feedGroupEntity: FeedGroupEntity): Int
+
+    @Query("DELETE FROM feed_group")
+    abstract fun deleteAll(): Int
+
+    @Query("DELETE FROM feed_group WHERE uid = :groupId")
+    abstract fun delete(groupId: Long): Int
+
+    @Query("SELECT subscription_id FROM feed_group_subscription_join WHERE group_id = :groupId")
+    abstract fun getSubscriptionIdsFor(groupId: Long): Flowable<List<Long>>
+
+    @Query("DELETE FROM feed_group_subscription_join WHERE group_id = :groupId")
+    abstract fun deleteSubscriptionsFromGroup(groupId: Long): Int
+
+    @Insert(onConflict = OnConflictStrategy.IGNORE)
+    abstract fun insertSubscriptionsToGroup(entities: List<FeedGroupSubscriptionEntity>): List<Long>
+
+    @Transaction
+    open fun updateSubscriptionsForGroup(groupId: Long, subscriptionIds: List<Long>) {
+        deleteSubscriptionsFromGroup(groupId)
+        insertSubscriptionsToGroup(subscriptionIds.map { FeedGroupSubscriptionEntity(groupId, it) })
+    }
+
+    @Transaction
+    open fun updateOrder(orderMap: Map<Long, Long>) {
+        orderMap.forEach { (groupId, sortOrder) -> updateOrder(groupId, sortOrder) }
+    }
+
+    @Query("UPDATE feed_group SET sort_order = :sortOrder WHERE uid = :groupId")
+    abstract fun updateOrder(groupId: Long, sortOrder: Long): Int
+
+    @Query("SELECT IFNULL(MAX(sort_order) + 1, 0) FROM feed_group")
+    protected abstract fun nextSortOrder(): Long
+
+    @Insert(onConflict = OnConflictStrategy.ABORT)
+    protected abstract fun insertInternal(feedGroupEntity: FeedGroupEntity): Long
+}

app/src/main/java/org/schabi/newpipe/database/feed/model/FeedEntity.kt

@@ -0,0 +1,43 @@
+package org.schabi.newpipe.database.feed.model
+
+import androidx.room.ColumnInfo
+import androidx.room.Entity
+import androidx.room.ForeignKey
+import androidx.room.Index
+import org.schabi.newpipe.database.feed.model.FeedEntity.Companion.FEED_TABLE
+import org.schabi.newpipe.database.feed.model.FeedEntity.Companion.STREAM_ID
+import org.schabi.newpipe.database.feed.model.FeedEntity.Companion.SUBSCRIPTION_ID
+import org.schabi.newpipe.database.stream.model.StreamEntity
+import org.schabi.newpipe.database.subscription.SubscriptionEntity
+
+@Entity(tableName = FEED_TABLE,
+        primaryKeys = [STREAM_ID, SUBSCRIPTION_ID],
+        indices = [Index(SUBSCRIPTION_ID)],
+        foreignKeys = [
+            ForeignKey(
+                    entity = StreamEntity::class,
+                    parentColumns = [StreamEntity.STREAM_ID],
+                    childColumns = [STREAM_ID],
+                    onDelete = ForeignKey.CASCADE, onUpdate = ForeignKey.CASCADE, deferred = true),
+            ForeignKey(
+                    entity = SubscriptionEntity::class,
+                    parentColumns = [SubscriptionEntity.SUBSCRIPTION_UID],
+                    childColumns = [SUBSCRIPTION_ID],
+                    onDelete = ForeignKey.CASCADE, onUpdate = ForeignKey.CASCADE, deferred = true)
+        ]
+)
+data class FeedEntity(
+        @ColumnInfo(name = STREAM_ID)
+        var streamId: Long,
+
+        @ColumnInfo(name = SUBSCRIPTION_ID)
+        var subscriptionId: Long
+) {
+
+    companion object {
+        const val FEED_TABLE = "feed"
+
+        const val STREAM_ID = "stream_id"
+        const val SUBSCRIPTION_ID = "subscription_id"
+    }
+}