Curl works but python requests doesnt (curl gives 304 and 200, requests a 403). Datadome, Cloudflare?

Question

When I go to vinted.nl a cookie is set. This allows me to do an api call in the browser and i can also do it in curl, but not using python requests. I am also wondering if this is because they use Datadome to prevent certain http requests? Or is it just a mistake in my python requests code? Or is it CloudFlare causing this?

I've tried VPN, home network, work network etc. Always this response when i use python requests, but when i use curl or postman i do get a correct response from the api.

This works:

curl 'https://www.vinted.nl/api/v2/languages' \
  -H 'cookie: anon_id=2ec2c37b-7d7a-4e65-b010-f9779852300c; v_udt=dFpFa0JZbDhLcnpXTzBMRGNHQ1Qwa3RGZm1vNFg0aVhOM29WTmc9PS0tVXIrVEtZRDNhMFdVMnhnbS0tVE1qVVVUeHRFMEk3LzVESlVwUEZ0UT09; v_sid=2e6ae0379a464a29cd97ead68aae344f; ab.optOut=This-cookie-will-expire-in-2024; _pbjs_userid_consent_data=7944749324711140; __cf_bm=dYupWIvIxLP6czwXium_JL.Sa1ALZDQydFWupYQS2lc-1689514518-0-Aev5HXwCkVwTwm5ecWnXi203ANDsvXiys1gB77le1YfP7/taQfDKbSTZadENhutigSewOGb98KaDPQrzXAIwJQYsYnIhckhdrhgaUdvJ3pkogtE6Jwxqiy2dZrbksApOrA==; viewport_size=374; _vinted_fr_session=b1psTUhzYXE0cjEwMGw4ck9CNWlVeEVtcG5kRTRES1dneHE0c3pEOEtxbUxNM0ZpTFdSS0NpZDdBdSs0a0FnR0RsZmtRL25wbDJBNjV0NFBKZzkwTWhBQzB4bndtNEhsWUFGVEpHM2xYRkY2K2ozcEhsUVNDdGgraHJBdlZVSVlRNUc0WHRHb0tidEFLQmI3NFovVEVTQUtDWnJvNkFKdlloVTRrL01UdjU3SW9vdWxmT3Y5RG12dTYvNGNMQmtxdkdyTHRMdUZuYWc0VW1vdGJYZGZydnFrbURUZFBHa2Z1b040VHM2WXpGT0dvbzJNZEFmY3FTYWVXcFR5OS9BYXZuMFU1NFpLTjY3WW83QlBlTjVNL0pDWm5WTXVoRmQ3Zm5Obmh3SjdsOTdrTU5rWjd5MGNydFlBVVJ1RmJjKzhFZkNEMy81MEoyeDB2Zm5kdGRMcmMzaElYSlkyNW9nTjZWRks2VXpjUVp2WW9TY3dDOWFtVnNNVDBabHhzZXhvZS9HQ2p4NTh6VHdKTWhCajVpSlRZQUlZdXBlZXVEUFlCaSt1ejU4U1pLUTNMMVNiTmR4ZEdBa1RKMDBLTUVJV0JrYkdmaStoN3BEcWNSWWgwdCttNE1Gd0FOSG9aeS9mRTFRZmhSaStZN3A5TXhiOXJjdE5LWWxPNzBBK29ydHl5MDhDbEd6RlU5OVJLYS95cVBNLzhnRWRjRlV0U00vL3RzYVZjdGVlY1dWTVl1cURLa1BjWnFabm5Ga01vUUNOYzMxUTRua3lyKyt3d2c1SlR5SU1KS3UvT3N3YUxsYzUzWURXQy95Q2V6TUpXeE1hdXk5eUR5dGFUSWZpYStLeEZLcU13NHBFcW9CT2d0VEZMWGZQbjdDMnFMV21FMk5WRmRxclhGYWlYaFNxa0lQeW9OYXpQSmpjVGQ5bGx5WmhaUGN0NjMxVmRNUDViTWE2ZnJ1bUlhNkh0R09FeCtJSCtrMERrTnVoRGFJWkhPc1JLa2x4QUlyb1dwYk1wSWJmNEM1ZGFSQlNVaGs2QndYRnBZZC90Z29ZNTZ2UXZVWWVHWFg1TVA3ZDY2RDFhbzJjKzVKMU04d0JyWEZQWUtsdlpFanAwRSt1R0FiR0EwTklCeDJyM0Mwc25LUktTMzRXNU1VaU45RGJweWdPNHJyUGx3UkxUenJ5Sy9PcndZb1JBemhqWUJUZTl4bGJadGtWUjEzSHRYNjRodkNETU9MS1ZTWXBkUTdQQ21OdDdTdWMvNzZLazRQY0ZiZlJpRXFjVHFTQWZJZ3B0RXdCS2dTTWRvb0J4RUFhTzZ2cFBKM0JpNFpNZnpValFYZ29va0FtRGgvZVc0VWhCNkd5aGFSVXJNVUZmRDEwVmNxbkMwU05QeW8vSXVpV3AvOER3cEd4Kzk5K0tISndNWkZJTzFpRGVGQXJVZmFxd1lpb1c0ZFllakUzYVNVQmswUDFzVFNRNmNDVkFYRUVRWjlPTnY2QXN0Zmw0MC94S3Ewd0JOR2gyNCtONjhQNnVtOXNodi9TbThLSUhhbEduM1h5N0IxZjg0UUh0NDFrMHhvejJkZUtKVHBUQ3hjdk1DST0tLVJnaVJLemdFamhpZ1pZQWphd3dpaUE9PQ%3D%3D--3c0df38decfa6be41b27f65b0f8073cf25f84a48; datadome=2~IXr6AtVrQ0uioqk1ZON1BWpEibj33CUCIOS8wuwAYvBWYjeE60K14p5Njb3e4VqYk2k3NOFnLFTDgRwSX4fC9dCfPRXpwgksoLXck2Y56oUZ2yK~nbLHpZdGyOnZh4; OptanonConsent=isGpcEnabled=0&datestamp=Sun+Jul+16+2023+15%3A49%3A51+GMT%2B0200+(Central+European+Summer+Time)&version=202305.1.0&browserGpcFlag=0&isIABGlobal=false&consentId=2ec2c37b-7d7a-4e65-b010-f9779852300c&hosts=&interactionCount=0&landingPath=https%3A%2F%2Fwww.vinted.nl%2F&groups=C0001%3A1%2CC0002%3A0%2CC0003%3A0%2CC0004%3A0%2CSTACK42%3A0%2CC0015%3A0; _dd_s=rum=0&expire=1689516288792'

I saw that curl sets some additional default headers, so i added those also to my python request:

> GET /api/v2/languages HTTP/2
> Host: www.vinted.nl
> user-agent: curl/7.88.1
> accept: */*
> cookie: anon_id=2ec2c37b-7d7a-4e65-b010-f9779852300c etc etc

However when i try this with python requests, I get a 403:

import requests

cookie = "anon_id=2ec2c37b-7d7a-4e65-b010-f9779852300c; v_udt=dFpFa0JZbDhLcnpXTzBMRGNHQ1Qwa3RGZm1vNFg0aVhOM29WTmc9PS0tVXIrVEtZRDNhMFdVMnhnbS0tVE1qVVVUeHRFMEk3LzVESlVwUEZ0UT09; v_sid=2e6ae0379a464a29cd97ead68aae344f; ab.optOut=This-cookie-will-expire-in-2024; _pbjs_userid_consent_data=7944749324711140; __cf_bm=dYupWIvIxLP6czwXium_JL.Sa1ALZDQydFWupYQS2lc-1689514518-0-Aev5HXwCkVwTwm5ecWnXi203ANDsvXiys1gB77le1YfP7/taQfDKbSTZadENhutigSewOGb98KaDPQrzXAIwJQYsYnIhckhdrhgaUdvJ3pkogtE6Jwxqiy2dZrbksApOrA==; viewport_size=374; _vinted_fr_session=b1psTUhzYXE0cjEwMGw4ck9CNWlVeEVtcG5kRTRES1dneHE0c3pEOEtxbUxNM0ZpTFdSS0NpZDdBdSs0a0FnR0RsZmtRL25wbDJBNjV0NFBKZzkwTWhBQzB4bndtNEhsWUFGVEpHM2xYRkY2K2ozcEhsUVNDdGgraHJBdlZVSVlRNUc0WHRHb0tidEFLQmI3NFovVEVTQUtDWnJvNkFKdlloVTRrL01UdjU3SW9vdWxmT3Y5RG12dTYvNGNMQmtxdkdyTHRMdUZuYWc0VW1vdGJYZGZydnFrbURUZFBHa2Z1b040VHM2WXpGT0dvbzJNZEFmY3FTYWVXcFR5OS9BYXZuMFU1NFpLTjY3WW83QlBlTjVNL0pDWm5WTXVoRmQ3Zm5Obmh3SjdsOTdrTU5rWjd5MGNydFlBVVJ1RmJjKzhFZkNEMy81MEoyeDB2Zm5kdGRMcmMzaElYSlkyNW9nTjZWRks2VXpjUVp2WW9TY3dDOWFtVnNNVDBabHhzZXhvZS9HQ2p4NTh6VHdKTWhCajVpSlRZQUlZdXBlZXVEUFlCaSt1ejU4U1pLUTNMMVNiTmR4ZEdBa1RKMDBLTUVJV0JrYkdmaStoN3BEcWNSWWgwdCttNE1Gd0FOSG9aeS9mRTFRZmhSaStZN3A5TXhiOXJjdE5LWWxPNzBBK29ydHl5MDhDbEd6RlU5OVJLYS95cVBNLzhnRWRjRlV0U00vL3RzYVZjdGVlY1dWTVl1cURLa1BjWnFabm5Ga01vUUNOYzMxUTRua3lyKyt3d2c1SlR5SU1KS3UvT3N3YUxsYzUzWURXQy95Q2V6TUpXeE1hdXk5eUR5dGFUSWZpYStLeEZLcU13NHBFcW9CT2d0VEZMWGZQbjdDMnFMV21FMk5WRmRxclhGYWlYaFNxa0lQeW9OYXpQSmpjVGQ5bGx5WmhaUGN0NjMxVmRNUDViTWE2ZnJ1bUlhNkh0R09FeCtJSCtrMERrTnVoRGFJWkhPc1JLa2x4QUlyb1dwYk1wSWJmNEM1ZGFSQlNVaGs2QndYRnBZZC90Z29ZNTZ2UXZVWWVHWFg1TVA3ZDY2RDFhbzJjKzVKMU04d0JyWEZQWUtsdlpFanAwRSt1R0FiR0EwTklCeDJyM0Mwc25LUktTMzRXNU1VaU45RGJweWdPNHJyUGx3UkxUenJ5Sy9PcndZb1JBemhqWUJUZTl4bGJadGtWUjEzSHRYNjRodkNETU9MS1ZTWXBkUTdQQ21OdDdTdWMvNzZLazRQY0ZiZlJpRXFjVHFTQWZJZ3B0RXdCS2dTTWRvb0J4RUFhTzZ2cFBKM0JpNFpNZnpValFYZ29va0FtRGgvZVc0VWhCNkd5aGFSVXJNVUZmRDEwVmNxbkMwU05QeW8vSXVpV3AvOER3cEd4Kzk5K0tISndNWkZJTzFpRGVGQXJVZmFxd1lpb1c0ZFllakUzYVNVQmswUDFzVFNRNmNDVkFYRUVRWjlPTnY2QXN0Zmw0MC94S3Ewd0JOR2gyNCtONjhQNnVtOXNodi9TbThLSUhhbEduM1h5N0IxZjg0UUh0NDFrMHhvejJkZUtKVHBUQ3hjdk1DST0tLVJnaVJLemdFamhpZ1pZQWphd3dpaUE9PQ%3D%3D--3c0df38decfa6be41b27f65b0f8073cf25f84a48; datadome=2~IXr6AtVrQ0uioqk1ZON1BWpEibj33CUCIOS8wuwAYvBWYjeE60K14p5Njb3e4VqYk2k3NOFnLFTDgRwSX4fC9dCfPRXpwgksoLXck2Y56oUZ2yK~nbLHpZdGyOnZh4; OptanonConsent=isGpcEnabled=0&datestamp=Sun+Jul+16+2023+15%3A49%3A51+GMT%2B0200+(Central+European+Summer+Time)&version=202305.1.0&browserGpcFlag=0&isIABGlobal=false&consentId=2ec2c37b-7d7a-4e65-b010-f9779852300c&hosts=&interactionCount=0&landingPath=https%3A%2F%2Fwww.vinted.nl%2F&groups=C0001%3A1%2CC0002%3A0%2CC0003%3A0%2CC0004%3A0%2CSTACK42%3A0%2CC0015%3A0; _dd_s=rum=0&expire=1689516288792"

headers = {
    "Host": "www.vinted.nl",
    "User-Agent": "curl/7.88.1",
    "Accept-Encoding": None,
    "Connection": None,
    "accept": "*/*",
    "cookie": cookie,
}
url = "https://www.vinted.nl/api/v2/languages"

response = requests.get(url=url, headers=headers)

print(response)

The cookie probably expires, so you probably have to get the cookie value from your browser developer tools if you want to try my code.

This is a screenshot of how i get the cookie using Chrome dev tools:

response.status_code: 403

response.text:

  <h1>Access denied</h1>
  <p>You do not have access to www.vinted.nl.</p><p>The site owner may have set restrictions that prevent you from accessing the site.</p>
  <ul class="cferror_details">
    <li>Ray ID: 7e90b3ed3aa9b8c6</li>
    <li>Timestamp: 2023-07-19 05:53:13 UTC</li>
    <li>Your IP address: 86.etc</li>
    <li class="XXX_no_wrap_overflow_hidden">Requested URL: www.vinted.nl/api/v2/languages </li>
    <li>Error reference number: 1020</li>
    <li>Server ID: FL_521F90</li>
    <li>User-Agent: Mozilla/5.0 (Linux; Android 8.0.0; SM-G955U Build/R16NW) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Mobile Safari/537.36</li>
  </ul>

response.headers

{'Date': 'Wed, 19 Jul 2023 05:53:13 GMT', 
'Content-Type': 'text/html; charset=UTF-8', 
'Transfer-Encoding': 'chunked', 
'Connection': 'keep-alive', 
'X-Frame-Options': 'SAMEORIGIN', 
'Referrer-Policy': 'same-origin', 
'Cache-Control': 'private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0', 'Expires': 'Thu, 01 Jan 1970 00:00:01 GMT', 
'Set-Cookie': '__cf_bm=LPhOJ80GUwCYRjFvS6m.8i.7NVl_YXcuEJufH5WHhrQ-1689745993-0-AWu4bkuqNL4hf1bx/IwLfM/RZY9dzgtyBMXq6m/gLv+s+nUHb1Lbc5Cw3/XPcVX9OnUAQURGbclo/OI7TyglbdEdVQr7S5r0Yq334L15kkKx; path=/; expires=Wed, 19-Jul-23 06:23:13 GMT; domain=.vinted.nl; HttpOnly; Secure; SameSite=None',
 'Vary': 'Accept-Encoding', 
'Server': 'cloudflare', 
'CF-RAY': '7e90b3ed3aa9b8c6-AMS', 
'Content-Encoding': 'gzip'}

response.request.headers:

{
'User-Agent': 'Mozilla/5.0 (Linux; Android 8.0.0; SM-G955U Build/R16NW) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Mobile Safari/537.36', 
'Accept-Encoding': 'gzip, deflate', 
'Accept': '*/*', 
'Connection': 'keep-alive', 
'host': 'www.vinted.nl', 
'Cache-Control': 'no-cache', 
'cookie': '_vinted_fr_session=MElmUmdUcmNOb etc etc'}

Where did you get the cookie in the first place? Is it only good for one request? (In that, you use it for `curl`, but try to reuse it in Python it fails.) What happens if you get a cookie and use it with Python first, then try to reuse it in `curl`? — chepner, Jul 16 '23 at 14:41
@chepner cookie can be used many times, i've added a screenshot of how i got it — Sander van den Oord, Jul 16 '23 at 15:14
@AlirezaRoshanzamir I've added the error response i get when using requests — Sander van den Oord, Jul 19 '23 at 06:01
Well that `
Access denied
` indicates a user-specific blockage. Probably your ip or browser did to much requests and is now blocked. Try it with a VPN so make sure it's not your curent network that's causing the issue. — 0stone0, Jul 19 '23 at 11:46
i actually tried with vpn, same result. i tried home network, work network. Why would curl and postman work, but not python requests? :( — Sander van den Oord, Jul 19 '23 at 11:47

score 2 · Answer 1 · answered Jul 18 '23 at 16:00

2

Steps I took to try/resolve this:

Copy the CURL as you stated in your screenshot
Imported the CURL command into Postman
Removed each cookie, send the request, received an invalid_authentication_token
Then, I've added each cookie 1 by 1 until I got a valid JSON respone.

The things I learned:

The only required cookie is: _vinted_fr_session
The API doesn't like an CURL user-agent, I suggest removing that from your code

I can get the expected JSON with the following Python code:

import requests

url = "https://www.vinted.nl/api/v2/languages"

cookie = '_vinted_fr_session=XXX; '

response = requests.request("GET", url, headers={
    "cookie": cookie
})

print(response.text)

Where XXX was in my case a 1763 long string, that I've copied from by Google Chrome Cookie.

The reply from vinted:

{
  "languages": [
    {
      "id": 32,
      "code": "nl-fr",
      "title": "Dutch",
      "title_local": "Nederlands",
      "current": false,
      "title_short": "NL"
    },
    {
      "id": 10,
      "code": "fr",
      "title": "French",
      "title_local": "Français",
      "current": true,
      "title_short": "FR"
    },
    {
      "id": 30,
      "code": "en-fr",
      "title": "English",
      "title_local": "English",
      "current": false,
      "title_short": "EN"
    },
    {
      "id": 31,
      "code": "es-fr",
      "title": "Spanish",
      "title_local": "Español",
      "current": false,
      "title_short": "ES"
    }
  ],
  "code": 0
}

answered Jul 18 '23 at 16:00

0stone0

34,288
4
39
64

Thanks for your explanation! I'm literally doing what you are doing, and i can do it with curl, but with python requests i get 403. Btw, python requests sets the user agent to User-Agent: python-requests/2.31.0. I also noticed that Postman actually changes the cookie that gets sent. Is this because of redirects? – Sander van den Oord Jul 18 '23 at 17:41
Don't think so since `curl 'http..' -H 'cookie: _vinted_fr_session'` also gives the correct response. If a redirect was needed, you'll need the `-L` option in CURL to enable it. So there must be something else going on with our python code. Some random thoughts: 1) Check the HTTP response code, it might be an ip issue. 2) NO firewall blocking stuff? – 0stone0 Jul 19 '23 at 09:58
yeah, i added the responses to the question, if you have any further ideas :) I tried the code on home network, and on work network, same result. – Sander van den Oord Jul 19 '23 at 11:43
Mmm no clue, can't get it reproduced, I even created a loop to call that simple curl 100 times and I wasn't blocked. It might be your user (/cookie), perhaps try a different account? Shooting blanks now ;) – 0stone0 Jul 19 '23 at 12:05
1

ok, i "solved" it now by creating a virtual env, now it works, also added it my "fix" as an answer. Thanks for all your help! Really helped checking what the problem could be and learning some new stuff along the way :) – Sander van den Oord Jul 19 '23 at 13:42

score 0 · Accepted Answer · answered Jul 19 '23 at 13:40

So it was:

not the vpn (ip),
not behind a firewall,
not work network vs home network,
not the specific headers etc,
not the cookie,
not the automatic headers that requests or curl sets,
not deleting all existing vinted.nl cookies.

Solved this by creating a virtual environment (using poetry) with a new python 3.11.4 and requests 2.31.0. After that it did all of a sudden work. Before this I was using the same python version and requests version, but the base installment. There must be something wrong there.

I was also starting to think in this direction, but who knows what the actual problem is (python >=3.6 on OSX has no certificates at all, and can't validate any SSL connection): https://stackoverflow.com/a/42334357/3489155

Could have also tried still to use urllib instead of requests as a fix.

score 0 · Answer 3 · answered Jul 24 '23 at 05:32

Actually this will not work using request, because obviously the website is using cloudflare , it can work if you can deobfuscate their javascript and solve challenge, but the easy way is to use undetected chrome driver to get it to work by using xhr request inside the browser

see here for more reference: https://checkforcloudflare.selesti.com/?q=https://www.vinted.nl/

Curl works but python requests doesnt (curl gives 304 and 200, requests a 403). Datadome, Cloudflare?

Access denied

3 Answers3