Sign in Sign up
JA · EN · FR
SecPulse
Categories
CVE-2021-41125

Information Disclosure in scrapy (CVE-2021-41125)

Summary

Vulnerability in scrapy (CVE-2021-41125). Confidential information can be exposed externally. Exploitable via HttpAuthMiddleware. Mitigation: upgrade to 2.5.1 or later.

Original description

The description as published by the source.

Impact

If you use HttpAuthMiddleware (i.e. the http_user and http_pass spider attributes) for HTTP authentication, all requests will expose your credentials to the request target.

This includes requests generated by Scrapy components, such as robots.txt requests sent by Scrapy when the ROBOTSTXT_OBEY setting is set to True, or as requests reached through redirects.

Patches

Upgrade to Scrapy 2.5.1 and use the new http_auth_domain spider attribute to control which domains are allowed to receive the configured HTTP authentication credentials.

If you are using Scrapy 1.8 or a lower version, and upgrading to Scrapy 2.5.1 is not an option, you may upgrade to Scrapy 1.8.1 instead.

Workarounds

If you cannot upgrade, set your HTTP authentication credentials on a per-request basis, using for example the w3lib.http.basic_auth_header function to convert your credentials into a value that you can assign to the Authorization header of your request, instead of defining your credentials globally using HttpAuthMiddleware.

For more information

If you have any questions or comments about this advisory:

Source: OSV.dev (Open Source Vulnerabilities) NVD GitHub Advisory OSV

Auto-generated summary

Assembled by SecPulse from the source data (CVSS, CWE, affected versions) using templates — no AI involved. Check the original description and references below for exact details.

A vulnerability tracked as CVE-2021-41125 has been found in scrapy.
Attackers can target a specific entry point like HttpAuthMiddleware over the network to misuse the product.
Confidential information can be exposed externally.
What to do: upgrade scrapy to 2.5.1 or later.
If unsure, ask your IT team or search "scrapy CVE-2021-41125" on the vendor's site.

CVE-2021-41125 (scrapy) — CWE-200
Attack vector: remote (network-reachable)
Attack surface: HttpAuthMiddleware / http_user / http_pass / robots.txt
Patched: 2.5.1 — apply immediately
Workaround: If you cannot upgrade, set your HTTP authentication credentials on a per-request basis, using for example the w3lib.http.basic_auth_header function to convert your cr
Plan: 1) Audit SBOM/dependencies, 2) Stage→prod upgrade, 3) Add WAF/proxy monitoring on affected endpoints, 4) Hunt IOCs in logs.
Refs: see the GHSA / vendor advisory / patched release linked on this page.

❓ What is the problem

Information disclosure (CWE-200) exists in scrapy. Attackers reach the vulnerable code path via HttpAuthMiddleware.

📍 Affected scope

Check the vendor advisory for the affected versions of scrapy. Attack surface: HttpAuthMiddleware / http_user / http_pass / robots.txt.

🔥 Severity

Severity: ?. Confidential information can be exposed externally.

🔧 How to fix

Update to 2.5.1.

🛡️ Workaround

Workaround: If you cannot upgrade, set your HTTP authentication credentials on a per-request basis, using for example the w3lib.http.basic_auth_header function to convert your cr

🔍 Detection

Search webserver/proxy logs for unusual request patterns matching this CVE's known IOCs. Run grep -r 'scrapy' . against your dependency files (package-lock.json, requirements.txt, go.sum) to find affected services.

Response actions

Concrete steps and command examples for SOC/SRE teams to execute in order

  1. 1
    Identify exposure identify 
    grep -r 'scrapy' . | grep -v node_modules

    リポジトリと本番環境の依存ファイル (package-lock.json / requirements.txt / go.sum / Gemfile.lock 等) で scrapy を grep し、稼働しているサービス・バージョンを把握する。

  2. 2
    Apply temporary workaround mitigate 
    If you cannot upgrade, set your HTTP authentication credentials on a per-request basis, using for example the [`w3lib.http.basic_auth_header`](https://w3lib.readthedocs.io/en/latest/w3lib.html#w3lib.http.basic_auth_header) function to convert your cr

    パッチが適用されるまでの応急処置として、If you cannot upgrade, set your HTTP authentication credentials on a per-request basis, using for example the w3lib.http.basic_auth_header function to convert your cr を実施。回避策の副作用 (機能低下) を確認した上で。

  3. 3
    Apply patch patch 
    Upgrade scrapy to 2.5.1

    ステージング環境で 2.5.1 に上げて回帰テスト → 本番反映。回帰テストはアプリの主要ハッピーパスを含めること。

  4. 4
    Post-deployment verification verify 
    Confirm patched version is live in production

    パッチ適用後、ステージングで PoC または同等の悪用パターンを再現して脆弱性が閉じたことを確認。本番ではアクセスログ・WAF でアラート再発が無いか継続監視。

Affected packages

PyPI scrapy
[{"type":"ECOSYSTEM","events":[{"introduced":"2.0.0"},{"fixed":"2.5.1"}]}]

Fix status by distribution

Fixed packages published by each distribution for this CVE

References

We use cookies for analytics and ad measurement. This setting applies to all Game co., Ltd. sites. Details