v3.10.5

3.10.5 (2024-06-12)

Bug Fixes

• allow creating new adaptive crawler instance without any parameters (9b7f595)
• declare missing peer dependencies in @crawlee/browser package (#2532) (3357c7f)
• fix detection of HTTP site when using the useState in adaptive crawler (#2530) (7e195c1)
• mark context.request.loadedUrl and id as required inside the request handler (#2531) (2b54660)

v3.10.4

3.10.4 (2024-06-11)

Bug Fixes

• add waitForAllRequestsToBeAdded option to enqueueLinks helper (925546b), closes #2318
• add missing useState implementation into crawling context (eec4a71)
• make crawler.log publicly accessible (#2526) (3e9e665)
• playwright: allow passing new context options in launchOptions on type level (0519d40), closes #1849
• respect crawler.log when creating child logger for Statistics (0a0d75d), closes #2412

v3.10.3

3.10.3 (2024-06-07)

Bug Fixes

• adaptive-crawler: log only once for the committed request handler execution (#2524) (533bd3f)
• increase timeout for retiring inactive browsers (#2523) (195f176)
• respect implicit router when no requestHandler is provided in AdaptiveCrawler (#2518) (31083aa)
• revert the scaling steps back to 5% (5bf32f8)

Features

• add waitForSelector context helper + parseWithCheerio in adaptive crawler (#2522) (6f88e73)
• log desired concurrency in the default status message (9f0b796)

v3.10.2

3.10.2 (2024-06-03)

Bug Fixes

• Autodetect sitemap filetype from content (#2497) (62a9f40), closes #2461
• improve fix for double extension in KVS with HTML files (#2505) (157927d), closes #2419

Features

• Loading sitemaps from string (#2496) (38ed0d6), closes #2460

v3.10.1

3.10.1 (2024-05-23)

Bug Fixes

• adjust URL_NO_COMMAS_REGEX regexp to allow single character hostnames (#2492) (ec802e8), closes #2487
• investigate and temp fix for possible 0-concurrency bug in RQv2 (#2494) (4ebe820)
• provide URLs to the error snapshot (#2482) (7f64145), closes /github.com/apify/apify-sdk-js/blob/master/packages/apify/src/key_value_store.ts#L25

v3.10.0

3.10.0 (2024-05-16)

Bug Fixes

• EnqueueStrategy.All erroring with links using unsupported protocols (#2389) (8db3908)
• conversion between tough cookies and browser pool cookies (#2443) (74f73ab)
• fire local SystemInfo events every second (#2454) (1fa9a66)
• use createSessionFunction when loading Session from persisted state (#2444) (3c56b4c)
• do not drop statistics on migration/resurrection/resume (#2462) (8ce7dd4)
• double tier decrement in tiered proxy (#2468) (3a8204b)
• Fixed double extension for screenshots (#2419) (e8b39c4), closes #1980
• malformed sitemap url when sitemap index child contains querystring (#2430) (e4cd41c)
• return true when robots.isAllowed returns undefined (#2439) (6f541f8), closes #2437
• sitemap content-type check breaks on content-type parameters (#2442) (db7d372)

Features

• add FileDownload "crawler" (#2435) (d73756b)
• implement ErrorSnapshotter for error context capture (#2332) (e861dfd), closes #2280
• make RequestQueue v2 the default queue, see more on Apify blog (#2390) (41ae8ab), closes #2388

Performance Improvements

• improve scaling based on memory (#2459) (2d5d443)
• optimize RequestList memory footprint (#2466) (12210bd)
• optimize adding large amount of requests via crawler.addRequests() (#2456) (6da86a8)

v3.9.2

3.9.2 (2024-04-17)

Bug Fixes

• break up growing stack in AutoscaledPool.notify (#2422) (6f2e6b0), closes #2421
• don't call notify in addRequests() (#2425) (c4d5446), closes #2421

Features

• sitemap: Support CDATA in sitemaps (#2424) (635f046)

v3.9.1

3.9.1 (2024-04-11)

Features

• browserPerProxy browser launch option (#2418) (df57b29)

v3.9.0

3.9.0 (2024-04-10)

Bug Fixes

• include actual key in error message of KVS' setValue (#2411) (9089bf1)
• notify autoscaled pool about newly added requests (#2400) (a90177d)
• puppeteer: allow passing networkidle to waitUntil in gotoExtended (#2399) (5d0030d), closes #2398
• sitemaps support application/xml (#2408) (cbcf47a)

Features

• createAdaptivePlaywrightRouter utility (#2415) (cee4778), closes #2407
• tieredProxyUrls for ProxyConfiguration (#2348) (5408c7f)
• better newUrlFunction for ProxyConfiguration (#2392) (330598b), closes #2348 #2065
• expand #shadow-root elements automatically in parseWithCheerio helper (#2396) (a05b3a9)

v3.8.2

3.8.2 (2024-03-21)

Bug Fixes

• core: solve possible dead locks in RequestQueueV2 (#2376) (ffba095)
• correctly report gzip decompression errors (#2368) (84a2f17)
• puppeteer: improve detection of older versions (98d4e86), closes #2370
• use 0 (number) instead of false as default for sessionRotationCount (#2372) (667a3e7)

Features

• implement global storage access checking and use it to prevent unwanted side effects in adaptive crawler (#2371) (fb3b7da), closes #2364