Merge pull request #312 from zeeshanakram3/proxy_rotation_improvements

Release: `v3.2.0`
Joystream · Dec 2, 2023 · 3e427c8 · 3e427c8
2 parents 5eb03e6 + f5aaf84
commit 3e427c8
Show file tree

Hide file tree

Showing 30 changed files with 560 additions and 181 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,3 +1,16 @@
+### 3.3.0
+
+- Added `chisel` proxy setup with automated IP rotation mechanism to circumvent youtube IP blockage, also integrated proxy setup with yt-synch server. See [docs](socks5-proxy/SETUP.md) for more details
+- Optimizes Youtube API quota usage by using `yt-dlp` instead of youtubeApi to fetch the channel details
+- Added `processedAt` in channels schema to track the timestamp when channel was processed (`Verified`/`Suspended`)
+- Removed unused method from video repository
+- Added YPP InductionRequirement of `CHANNEL_CRITERIA_UNMET_CREATION_DATE`
+- Update video processing priority calculation to prioritize Silver+ tier channels
+- **FIX**: Update referral reward values
+- **FIX**: Added error handling for winston logger
+- **FIX**: Remove stale downloaded videos
+- **FIX**: Avoid updating some channel properties e.g. `createdAt`, `email`, `userId` on re-signup
+
 ### 3.2.0
 
 - Optimizes Youtube API quota usage by using `yt-dlp` instead of youtubeApi to fetch the video details

diff --git a/config.yml b/config.yml
@@ -38,6 +38,10 @@ youtube:
   clientSecret: google-client-secret
   # adcKeyFilePath: path/to/adc-key-file.json
   # maxAllowedQuotaUsageInPercentage: 95
+proxy:
+  url: socks5://chisel-client:1080
+  chiselProxy:
+    ec2AutoRotateIp: false
 aws:
   endpoint: http://localhost:4566
   region: us-east-1

diff --git a/docker-compose.yml b/docker-compose.yml
@@ -43,12 +43,16 @@ services:
       # YT_SYNCH__JOYSTREAM__FAUCET__CAPTCHA_BYPASS_KEY: ${YT_SYNCH__JOYSTREAM__FAUCET__CAPTCHA_BYPASS_KEY}
     ports:
       - 127.0.0.1:${YT_SYNCH__HTTP_API__PORT}:${YT_SYNCH__HTTP_API__PORT}
+    networks:
+      - youtube-synch
   redis:
     image: redis:7.2.1
     container_name: redis
     command: [ "redis-server", "--maxmemory-policy", "noeviction" ]
     ports:
       - 127.0.0.1:${YT_SYNCH__ENDPOINTS__REDIS__PORT}:${YT_SYNCH__ENDPOINTS__REDIS__PORT}
+    networks:
+      - youtube-synch
     volumes:
       - redis-data:/data
 
@@ -59,3 +63,7 @@ volumes:
     driver: local
   redis-data:
     driver: local
+
+networks:
+  youtube-synch:
+    name: youtube-synch
diff --git a/...t-configuration-used-by-yt-dlp-to-bypass-ip-blockage-by-youtube-dependencies.md b/...t-configuration-used-by-yt-dlp-to-bypass-ip-blockage-by-youtube-dependencies.md
@@ -0,0 +1,3 @@
+## dependencies Type
+
+unknown
diff --git a/...ass-ip-blockage-by-youtube-properties-chiselproxy-properties-ec2autorotateip.md b/...ass-ip-blockage-by-youtube-properties-chiselproxy-properties-ec2autorotateip.md
@@ -0,0 +1,3 @@
+## ec2AutoRotateIp Type
+
+`boolean`
diff --git a/...ation-used-by-yt-dlp-to-bypass-ip-blockage-by-youtube-properties-chiselproxy.md b/...ation-used-by-yt-dlp-to-bypass-ip-blockage-by-youtube-properties-chiselproxy.md
@@ -0,0 +1,27 @@
+## chiselProxy Type
+
+`object` ([Details](definition-properties-socks5-proxy-client-configuration-used-by-yt-dlp-to-bypass-ip-blockage-by-youtube-properties-chiselproxy.md))
+
+# chiselProxy Properties
+
+| Property                            | Type      | Required | Nullable       | Defined by                                                                                                                                                                                                                                                                                                               |
+| :---------------------------------- | :-------- | :------- | :------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [ec2AutoRotateIp](#ec2autorotateip) | `boolean` | Optional | cannot be null | [Youtube Sync node configuration](definition-properties-socks5-proxy-client-configuration-used-by-yt-dlp-to-bypass-ip-blockage-by-youtube-properties-chiselproxy-properties-ec2autorotateip.md "https://joystream.org/schemas/youtube-synch/config#/properties/proxy/properties/chiselProxy/properties/ec2AutoRotateIp") |
+
+## ec2AutoRotateIp
+
+Boolean option to enable auto rotation of ec2 instance IP where chisel server is running by restating it
+
+`ec2AutoRotateIp`
+
+*   is optional
+
+*   Type: `boolean`
+
+*   cannot be null
+
+*   defined in: [Youtube Sync node configuration](definition-properties-socks5-proxy-client-configuration-used-by-yt-dlp-to-bypass-ip-blockage-by-youtube-properties-chiselproxy-properties-ec2autorotateip.md "https://joystream.org/schemas/youtube-synch/config#/properties/proxy/properties/chiselProxy/properties/ec2AutoRotateIp")
+
+### ec2AutoRotateIp Type
+
+`boolean`
diff --git a/...configuration-used-by-yt-dlp-to-bypass-ip-blockage-by-youtube-properties-url.md b/...configuration-used-by-yt-dlp-to-bypass-ip-blockage-by-youtube-properties-url.md
@@ -0,0 +1,3 @@
+## url Type
+
+`string`
diff --git a/...5-proxy-client-configuration-used-by-yt-dlp-to-bypass-ip-blockage-by-youtube.md b/...5-proxy-client-configuration-used-by-yt-dlp-to-bypass-ip-blockage-by-youtube.md
@@ -0,0 +1,46 @@
+## proxy Type
+
+`object` ([Socks5 proxy client configuration used by yt-dlp to bypass IP blockage by Youtube](definition-properties-socks5-proxy-client-configuration-used-by-yt-dlp-to-bypass-ip-blockage-by-youtube.md))
+
+# proxy Properties
+
+| Property                    | Type     | Required | Nullable       | Defined by                                                                                                                                                                                                                                                         |
+| :-------------------------- | :------- | :------- | :------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| [url](#url)                 | `string` | Optional | cannot be null | [Youtube Sync node configuration](definition-properties-socks5-proxy-client-configuration-used-by-yt-dlp-to-bypass-ip-blockage-by-youtube-properties-url.md "https://joystream.org/schemas/youtube-synch/config#/properties/proxy/properties/url")                 |
+| [chiselProxy](#chiselproxy) | `object` | Optional | cannot be null | [Youtube Sync node configuration](definition-properties-socks5-proxy-client-configuration-used-by-yt-dlp-to-bypass-ip-blockage-by-youtube-properties-chiselproxy.md "https://joystream.org/schemas/youtube-synch/config#/properties/proxy/properties/chiselProxy") |
+
+## url
+
+Proxy Client URL e.g. socks\://localhost:1080, socks\://user:password\@localhost:1080
+
+`url`
+
+*   is optional
+
+*   Type: `string`
+
+*   cannot be null
+
+*   defined in: [Youtube Sync node configuration](definition-properties-socks5-proxy-client-configuration-used-by-yt-dlp-to-bypass-ip-blockage-by-youtube-properties-url.md "https://joystream.org/schemas/youtube-synch/config#/properties/proxy/properties/url")
+
+### url Type
+
+`string`
+
+## chiselProxy
+
+Configuration option to manage Chisel Client & Server. Before enabling this option please refer to setup guide in `socks5-proxy/SETUP.md`
+
+`chiselProxy`
+
+*   is optional
+
+*   Type: `object` ([Details](definition-properties-socks5-proxy-client-configuration-used-by-yt-dlp-to-bypass-ip-blockage-by-youtube-properties-chiselproxy.md))
+
+*   cannot be null
+
+*   defined in: [Youtube Sync node configuration](definition-properties-socks5-proxy-client-configuration-used-by-yt-dlp-to-bypass-ip-blockage-by-youtube-properties-chiselproxy.md "https://joystream.org/schemas/youtube-synch/config#/properties/proxy/properties/chiselProxy")
+
+### chiselProxy Type
+
+`object` ([Details](definition-properties-socks5-proxy-client-configuration-used-by-yt-dlp-to-bypass-ip-blockage-by-youtube-properties-chiselproxy.md))
diff --git a/...nition-properties-yt-synch-syncronization-related-settings-properties-limits.md b/...nition-properties-yt-synch-syncronization-related-settings-properties-limits.md
@@ -6,26 +6,26 @@
 
 | Property                                                | Type      | Required | Nullable       | Defined by                                                                                                                                                                                                                                                                                                                        |
 | :------------------------------------------------------ | :-------- | :------- | :------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| [dailyApiQuota](#dailyapiquota)                         | `object`  | Required | cannot be null | [Youtube Sync node configuration](definition-properties-yt-synch-syncronization-related-settings-properties-limits-properties-specifies-daily-youtube-api-quota-rationing-scheme-for-youtube-partner-program.md 'https://joystream.org/schemas/youtube-synch/config#/properties/sync/properties/limits/properties/dailyApiQuota') |
-| [maxConcurrentDownloads](#maxconcurrentdownloads)       | `number`  | Required | cannot be null | [Youtube Sync node configuration](definition-properties-yt-synch-syncronization-related-settings-properties-limits-properties-maxconcurrentdownloads.md 'https://joystream.org/schemas/youtube-synch/config#/properties/sync/properties/limits/properties/maxConcurrentDownloads')                                                |
-| [createVideoTxBatchSize](#createvideotxbatchsize)       | `number`  | Required | cannot be null | [Youtube Sync node configuration](definition-properties-yt-synch-syncronization-related-settings-properties-limits-properties-createvideotxbatchsize.md 'https://joystream.org/schemas/youtube-synch/config#/properties/sync/properties/limits/properties/createVideoTxBatchSize')                                                |
-| [maxConcurrentUploads](#maxconcurrentuploads)           | `number`  | Required | cannot be null | [Youtube Sync node configuration](definition-properties-yt-synch-syncronization-related-settings-properties-limits-properties-maxconcurrentuploads.md 'https://joystream.org/schemas/youtube-synch/config#/properties/sync/properties/limits/properties/maxConcurrentUploads')                                                    |
-| [pendingDownloadTimeoutSec](#pendingdownloadtimeoutsec) | `integer` | Required | cannot be null | [Youtube Sync node configuration](definition-properties-yt-synch-syncronization-related-settings-properties-limits-properties-pendingdownloadtimeoutsec.md 'https://joystream.org/schemas/youtube-synch/config#/properties/sync/properties/limits/properties/pendingDownloadTimeoutSec')                                          |
-| [storage](#storage)                                     | `string`  | Required | cannot be null | [Youtube Sync node configuration](definition-properties-yt-synch-syncronization-related-settings-properties-limits-properties-storage.md 'https://joystream.org/schemas/youtube-synch/config#/properties/sync/properties/limits/properties/storage')                                                                              |
+| [dailyApiQuota](#dailyapiquota)                         | `object`  | Required | cannot be null | [Youtube Sync node configuration](definition-properties-yt-synch-syncronization-related-settings-properties-limits-properties-specifies-daily-youtube-api-quota-rationing-scheme-for-youtube-partner-program.md "https://joystream.org/schemas/youtube-synch/config#/properties/sync/properties/limits/properties/dailyApiQuota") |
+| [maxConcurrentDownloads](#maxconcurrentdownloads)       | `number`  | Required | cannot be null | [Youtube Sync node configuration](definition-properties-yt-synch-syncronization-related-settings-properties-limits-properties-maxconcurrentdownloads.md "https://joystream.org/schemas/youtube-synch/config#/properties/sync/properties/limits/properties/maxConcurrentDownloads")                                                |
+| [createVideoTxBatchSize](#createvideotxbatchsize)       | `number`  | Required | cannot be null | [Youtube Sync node configuration](definition-properties-yt-synch-syncronization-related-settings-properties-limits-properties-createvideotxbatchsize.md "https://joystream.org/schemas/youtube-synch/config#/properties/sync/properties/limits/properties/createVideoTxBatchSize")                                                |
+| [maxConcurrentUploads](#maxconcurrentuploads)           | `number`  | Required | cannot be null | [Youtube Sync node configuration](definition-properties-yt-synch-syncronization-related-settings-properties-limits-properties-maxconcurrentuploads.md "https://joystream.org/schemas/youtube-synch/config#/properties/sync/properties/limits/properties/maxConcurrentUploads")                                                    |
+| [pendingDownloadTimeoutSec](#pendingdownloadtimeoutsec) | `integer` | Required | cannot be null | [Youtube Sync node configuration](definition-properties-yt-synch-syncronization-related-settings-properties-limits-properties-pendingdownloadtimeoutsec.md "https://joystream.org/schemas/youtube-synch/config#/properties/sync/properties/limits/properties/pendingDownloadTimeoutSec")                                          |
+| [storage](#storage)                                     | `string`  | Required | cannot be null | [Youtube Sync node configuration](definition-properties-yt-synch-syncronization-related-settings-properties-limits-properties-storage.md "https://joystream.org/schemas/youtube-synch/config#/properties/sync/properties/limits/properties/storage")                                                                              |
 
 ## dailyApiQuota
 
 Specifies daily Youtube API quota rationing scheme for Youtube Partner Program
 
 `dailyApiQuota`
 
-- is required
+*   is required
 
-- Type: `object` ([Specifies daily Youtube API quota rationing scheme for Youtube Partner Program](definition-properties-yt-synch-syncronization-related-settings-properties-limits-properties-specifies-daily-youtube-api-quota-rationing-scheme-for-youtube-partner-program.md))
+*   Type: `object` ([Specifies daily Youtube API quota rationing scheme for Youtube Partner Program](definition-properties-yt-synch-syncronization-related-settings-properties-limits-properties-specifies-daily-youtube-api-quota-rationing-scheme-for-youtube-partner-program.md))
 
-- cannot be null
+*   cannot be null
 
-- defined in: [Youtube Sync node configuration](definition-properties-yt-synch-syncronization-related-settings-properties-limits-properties-specifies-daily-youtube-api-quota-rationing-scheme-for-youtube-partner-program.md 'https://joystream.org/schemas/youtube-synch/config#/properties/sync/properties/limits/properties/dailyApiQuota')
+*   defined in: [Youtube Sync node configuration](definition-properties-yt-synch-syncronization-related-settings-properties-limits-properties-specifies-daily-youtube-api-quota-rationing-scheme-for-youtube-partner-program.md "https://joystream.org/schemas/youtube-synch/config#/properties/sync/properties/limits/properties/dailyApiQuota")
 
 ### dailyApiQuota Type
 
@@ -37,13 +37,13 @@ Max no. of videos that should be concurrently downloaded from Youtube to be prep
 
 `maxConcurrentDownloads`
 
-- is required
+*   is required
 
-- Type: `number`
+*   Type: `number`
 
-- cannot be null
+*   cannot be null
 
-- defined in: [Youtube Sync node configuration](definition-properties-yt-synch-syncronization-related-settings-properties-limits-properties-maxconcurrentdownloads.md 'https://joystream.org/schemas/youtube-synch/config#/properties/sync/properties/limits/properties/maxConcurrentDownloads')
+*   defined in: [Youtube Sync node configuration](definition-properties-yt-synch-syncronization-related-settings-properties-limits-properties-maxconcurrentdownloads.md "https://joystream.org/schemas/youtube-synch/config#/properties/sync/properties/limits/properties/maxConcurrentDownloads")
 
 ### maxConcurrentDownloads Type
 
@@ -63,13 +63,13 @@ No. of videos that should be created in a batched 'create_video' tx
 
 `createVideoTxBatchSize`
 
-- is required
+*   is required
 
-- Type: `number`
+*   Type: `number`
 
-- cannot be null
+*   cannot be null
 
-- defined in: [Youtube Sync node configuration](definition-properties-yt-synch-syncronization-related-settings-properties-limits-properties-createvideotxbatchsize.md 'https://joystream.org/schemas/youtube-synch/config#/properties/sync/properties/limits/properties/createVideoTxBatchSize')
+*   defined in: [Youtube Sync node configuration](definition-properties-yt-synch-syncronization-related-settings-properties-limits-properties-createvideotxbatchsize.md "https://joystream.org/schemas/youtube-synch/config#/properties/sync/properties/limits/properties/createVideoTxBatchSize")
 
 ### createVideoTxBatchSize Type
 
@@ -89,13 +89,13 @@ Max no. of videos that should be concurrently uploaded to Joystream's storage no
 
 `maxConcurrentUploads`
 
-- is required
+*   is required
 
-- Type: `number`
+*   Type: `number`
 
-- cannot be null
+*   cannot be null
 
-- defined in: [Youtube Sync node configuration](definition-properties-yt-synch-syncronization-related-settings-properties-limits-properties-maxconcurrentuploads.md 'https://joystream.org/schemas/youtube-synch/config#/properties/sync/properties/limits/properties/maxConcurrentUploads')
+*   defined in: [Youtube Sync node configuration](definition-properties-yt-synch-syncronization-related-settings-properties-limits-properties-maxconcurrentuploads.md "https://joystream.org/schemas/youtube-synch/config#/properties/sync/properties/limits/properties/maxConcurrentUploads")
 
 ### maxConcurrentUploads Type
 
@@ -115,13 +115,13 @@ Timeout for pending youtube video downloads in seconds
 
 `pendingDownloadTimeoutSec`
 
-- is required
+*   is required
 
-- Type: `integer`
+*   Type: `integer`
 
-- cannot be null
+*   cannot be null
 
-- defined in: [Youtube Sync node configuration](definition-properties-yt-synch-syncronization-related-settings-properties-limits-properties-pendingdownloadtimeoutsec.md 'https://joystream.org/schemas/youtube-synch/config#/properties/sync/properties/limits/properties/pendingDownloadTimeoutSec')
+*   defined in: [Youtube Sync node configuration](definition-properties-yt-synch-syncronization-related-settings-properties-limits-properties-pendingdownloadtimeoutsec.md "https://joystream.org/schemas/youtube-synch/config#/properties/sync/properties/limits/properties/pendingDownloadTimeoutSec")
 
 ### pendingDownloadTimeoutSec Type
 
@@ -137,24 +137,24 @@ Maximum total size of all downloaded assets stored in `downloadsDir`
 
 `storage`
 
-- is required
+*   is required
 
-- Type: `string`
+*   Type: `string`
 
-- cannot be null
+*   cannot be null
 
-- defined in: [Youtube Sync node configuration](definition-properties-yt-synch-syncronization-related-settings-properties-limits-properties-storage.md 'https://joystream.org/schemas/youtube-synch/config#/properties/sync/properties/limits/properties/storage')
+*   defined in: [Youtube Sync node configuration](definition-properties-yt-synch-syncronization-related-settings-properties-limits-properties-storage.md "https://joystream.org/schemas/youtube-synch/config#/properties/sync/properties/limits/properties/storage")
 
 ### storage Type
 
 `string`
 
 ### storage Constraints
 
-**pattern**: the string must match the following regular expression:
+**pattern**: the string must match the following regular expression: 
 
 ```regexp
 ^[0-9]+(B|K|M|G|T)$
 ```
 
-[try pattern](<https://regexr.com/?expression=%5E%5B0-9%5D%2B(B%7CK%7CM%7CG%7CT)%24> 'try regular expression with regexr.com')
+[try pattern](https://regexr.com/?expression=%5E%5B0-9%5D%2B\(B%7CK%7CM%7CG%7CT\)%24 "try regular expression with regexr.com")