I'm trying to automate the looking up of prices on the website cardmarket.com. I thought I could simply use python's requests, but the HTML I get is totally different from what shows in my webbrowser. This is my code:
import requests
from bs4 import BeautifulSoup as bs
card = 'sol+ring'
url = f'https://www.cardmarket.com/en/Magic/Products/Search?idCategory=0&idExpansion=0&searchString=%5B{card}%5D&exactMatch=on&onlyAvailable=on&idRarity=0&sortBy=price_asc&perSite=1'
html = requests.get(url).text
soup = bs(html, 'html.parser')
print(soup.prettify())
This returns:
> <!DOCTYPE html> <html lang="en-US"> <head> <title> Just a
> moment... </title> <meta content="text/html; charset=utf-8"
> http-equiv="Content-Type"/> <meta content="IE=Edge"
> http-equiv="X-UA-Compatible"/> <meta content="noindex,nofollow"
> name="robots"/> <meta content="width=device-width,initial-scale=1"
> name="viewport"/> <link href="/cdn-cgi/styles/challenges.css"
> rel="stylesheet"/> </head> <body class="no-js"> <div
> class="main-wrapper" role="main"> <div class="main-content">
> <noscript>
> <div id="challenge-error-title">
> <div class="h2">
> <span class="icon-wrapper">
> <div class="heading-icon warning-icon">
> </div>
> </span>
> <span id="challenge-error-text">
> Enable JavaScript and cookies to continue
> </span>
> </div>
> </div>
> </noscript>
> <div id="trk_jschal_js" style="display:none;background-image:url('/cdn-cgi/images/trace/managed/nojs/transparent.gif?ray=7c2340c85dc337f2')">
> </div>
> <form action="/en/Magic/Products/Search?idCategory=0&idExpansion=0&searchString=%5Bsol+ring%5D&exactMatch=on&onlyAvailable=on&idRarity=0&sortBy=price_asc&perSite=1&__cf_chl_f_tk=Y.l_DDIwIQE4RfQbth8EL91OZUHtyRmRaw_VKrnp_II-1683229620-0-gaNycGzNC-U" enctype="application/x-www-form-urlencoded" id="challenge-form"
> method="POST">
> <input name="md" type="hidden" value="am7bUf4.zdl4ChvYGJhF6.M4NncavvH4k1y3SWcjY40-1683229620-0-AZZwgHuQMPW_IbDZIr7VwvaK00tcVHHxq63Ae1132gHacl26HY1w9DWv_VSrApE0ORx4MEIuVw725CdeFGXb4ti-UJyQTbbMDoTZK2VoyJpEBka4I0Zxyk3LJ4cg7iQ9itCgqiNVotkMHLUOlNcgpralqPBiuk26ypioPZOpwqCXBM2J53Gub6sy_yUA_bXbMjPwNak-ggUcUgDj_lPnG7dkWYtHnoZU07PBtLcTRJH1bmI0zNOj32J7igDIHxxzolLtrVTcjv0u9mWFnQRu6hC-cjOcYzWoEBpdoFQvgy1QgcX9NOeQuGGp0bISvPn3eqrDDUd7qoPY-ds5TfyW7R0zzZCv8LWLmN3e33_tldzxO9J7CcO7Py16yoVxwDdsyS8eOq2oJT---2ydQyPDB_UNZneEb6lvKg2BtceEW1KUP1PLm-SfXqA6ri-KtLKuo0dC0ukZu8OGC3X2EF-_SHsbg8nDalCgqklgkVVqIC4ncWyzQafDqMEqOJf7Tbfy2pZ8N1Lh7XQKIXYp_4BsytNRRfNLlVO21KCWojaZKO4i379GAkWbDH-cPnT_pgYpFiOiX7FZmZUCmXrS8q7AaQIi9VHTlJLEim4jf7VXPZtsfeMB-gR6l8K5fkfuHcgFNoS0rFZ98dC0MQYLszDLHtstgEwjEWOVNJvqQhDS-6V1PnT6H6JSHeWZSbt6lHbN7LH3gbQx2eQSEh-Vw38BQOXWjS376esj9Q82pnfXdLhP4VO-nnDfUstsp3i58HhOvRbm83u5QhF7nkaJV0VntAXiJ9uMuQMEjMpB6J2u5hl5IT6dhV_vI4a-88EhiP_n4fjEVRaoNHrIhutZcMsJ_sFcX5rBa60Iah4X5Mrr58-O1JpCHcbEvF0AGB8JjzOwsXRuISnU_gVZ9ipJk6FOj-aUL5PqSc-u-9pe4EnV0MaRHBbt9MPcLdtQnga29efk5zFW-r8p92e8sCdwLQq5A4LmnjVUnBl5WbXLj-HMlyWJX8p4bRhhyhJUtITepDn25ioLj5SD4Hd0VrZElKLnm6YxGRpb8eJWRvbGEq-ZBZ_kGOLtNOhAZZF8FH2_rx8Rf5HY0z8lP3RJgQtC0KjqI-t35FT_ObXhdnVbie2VS4wzqkjPbl80jZBtpnj5_ZuM2FIHYmAX_9yQ76NhKbCnZMS7XGSArZKtV_4rxX3BAZwCrGwuk_cK_gdNp3722Qlfd0lkwFVtmQQ7lURF-PRkcyv2Q-fpm_VlYKIdMLY57gEhLx-V0MMBfApizHqjqhr_dLbo67erW3sZUG5pEBjBHlSCtI3EhunhevF3u7PiQNhKMzN7AnF6D_pFQNv5ylCs8nkGLSLIyFMb0-bL3e3hrDiCeRae6K8oF4bx1rV2IFejUU1zwBTm5MIf8NyWXtGfwO5jw89VUmp5f_JvBlJM_susCUMPxtrCxqlzAqXBretCtFX3hVBJJfCtzcgYQXVqLOzIkh75mcmrhQExO06I39dlUksZ172xzLmQ91TWf56HJyCQKjQkW5gcg9pnqx8q9DF6Sp4IH_QhdfYLpHuSoy6635qsBW0oVAslcYWidqD1CqZSpuPcFVBd_4vcple3u13ehtFSiC03txwBJ2sUTgbhRSVYJZkEGcph5CiapSMjU1jVS6931XcLlTQVdXZuPovOmuGA7lOBoFfEoQUGOSfoCQ609hsJqQbGKCV5RtZ4ma1XCNZ9LHrId93zfIhTwnNf7vJ4SHo4CUwI1LW5W3ie2P7B42qTQNWt2KQluSNqxF39OiFNC8oQer4aMhmzZGy9fKOw4tL3waEIkZe94jvx65Gb70chWUjqinHEHZufzryTpQys6qWurtf6iIjKrCNJKVvXlgtzDDNpZrk9AiI--xF2zrC4qpnF5iYeeMUXf8uHXdfmtl-eLN7Dd1iheKO23ceVI-drP-DETwyly9Hv_yZv_ceVO6Rk5O-GQXkdEhQVwCy1Oqn-1lFSVyppfrrxFM8WVXeeIrSUBiT_bvELaUEeWRSlwYI4vBQ5sLhr-eLNwOWASmMRce-3D2O0MfRFFxfAdVgqDARDEjXVKQROSRpfyUu6BmyLfrl0Hylz_CCmI_rMZDorcR6GVmUu8oHcqW6soP-__CBnH2JX-e3_fP1cmUDkmsll8Cla3LKZuo9HDzV1zhHY85OMCseRczmYAD78Xxb4T7oN4p4vh_CvTa8YJyYzl4xm69GmUo3cIW1_rVr9pxikLYsuum11AL09b49c-HejHprTjR04rFhsfbsKKEEvqDo92JFaq2pat26NN6xo4U_TyK7y-7Zc1s8kcfbouQO8V7I-VK-tK4ocxhnYuKXG9mAp3Ft6La8_gbv09y5YJrRXuk4wDojD_HYJlgVIlF1dSvziWeXC4BodxQbKbkUiT1INycsRVOppcaqv4hRtGSMywygFh7hGtBj_UIyMBJMP0p6ReP-A0R7GXmTYTCpu4JMwh2jVTT2_jLwPm0HSkpwAAO4DkynId3iiMOYQgNRjjGCRfiXnAUDtZTzcYliPZsOm1bVO4QY321flZA_DVBoB80yxgfqxUzJcTnFIS0oOUyZ_VVnr-_fAp9vnIVtd-UMGBsKjSlBtf3FbHPDI7y6SDwxrLzv4l0sf2x7F5pN9utJjdtzOu5-aolnzeLzKyZY9i0wb6eR7fCjvPmCbdNe0NbDnCfyHMVX6_pgvyRtO2lyI7NhdIIiKQj5sD5RhPtD34b4ySU1Lu0PXyTvlwKd05o72rWbT5aGcJzqL48kRuNteR8m3nWNo68F9sEDSNkMXXj7NaSBG_G_CDOgmZqNjw-gxFXYUTanE33coKwOGETpj3Z9v6Rr7fLA6saoN--jScaxedhe53kJQgJ5bwo-TiUn8yrTXQpGXiab0jPwFYk-B9nbEyYyICaNWgrdVmdFhuJC8HZnQ_iAdiRgUp0MyntSGu1uUNDqhYLsz70X6GJMSodmeGtyIRLxBjLoRNKGGbou0eJpY"/>
> </form> </div> </div> <script> (function(){
> window._cf_chl_opt={
> cvId: '2',
> cZone: 'www.cardmarket.com',
> cType: 'managed',
> cNounce: '59216',
> cRay: '7c2340c85dc337f2',
> cHash: '52201252631573f',
> cUPMDTk: "\/en\/Magic\/Products\/Search?idCategory=0&idExpansion=0&searchString=%5Bsol+ring%5D&exactMatch=on&onlyAvailable=on&idRarity=0&sortBy=price_asc&perSite=1&__cf_chl_tk=Y.l_DDIwIQE4RfQbth8EL91OZUHtyRmRaw_VKrnp_II-1683229620-0-gaNycGzNC-U",
> cFPWv: 'g',
> cTTimeMs: '1000',
> cMTimeMs: '0',
> cTplV: 5,
> cTplB: 'cf',
> cK: "",
> cRq: {
> ru: 'aHR0cHM6Ly93d3cuY2FyZG1hcmtldC5jb20vZW4vTWFnaWMvUHJvZHVjdHMvU2VhcmNoP2lkQ2F0ZWdvcnk9MCZpZEV4cGFuc2lvbj0wJnNlYXJjaFN0cmluZz0lNUJzb2wrcmluZyU1RCZleGFjdE1hdGNoPW9uJm9ubHlBdmFpbGFibGU9b24maWRSYXJpdHk9MCZzb3J0Qnk9cHJpY2VfYXNjJnBlclNpdGU9MQ==',
> ra: 'cHl0aG9uLXJlcXVlc3RzLzIuMjguMQ==',
> rm: 'R0VU',
> d: 'WLvKTceTT+DuPR8uR948I8oL77i021rpJCLo0JfsWXAVMBEzqjPF/jdWFbX/rDgR0cvGpxYh3cN0eYKLmmhgoI7MBHdtTPyG+GTjmCbf3ukbooQZ/NJOEL+rbfk8TF9nyrLAXbtngEgWzaj+LKbAUDP70MYFzDKWqHBBeN5kNyUSV84qVyNkc66zfQzkRJmgQOoSVP+U7i//debGl8DKED3B//eFnS8a3yLqTLdaGpY+32rMnWEvrtTRQ4HD8xnP5Ndwl6bl8/2Lwsq/8wTU9+GGmQ+Y6tXYly7ER4GAtojHEJcOw48AxzUfHgGN9Ri4NONX2RUzI+lKqocqSwSOiN1y97105KK6ky/v/Hq9y7t7K8gEhev+HbSouCzZN1VA0tprvqFwZ/1XfaEOd37lA3JawQrM2KTe026qS3CeqlJvmG3Xn+VcpYLCePIW5nUyGnTYP9MoaLcPG8S/l4gKX8eXuPkTFhxfRLCKw7gdLqjGOqNnNrEhFzs6UhfU1RCJzDIhlc8rnJYYJVcsSv0ZI2ihkAL6UHGLdH4K7pWvNsL63JlT5GzZQfztvDjr6a18Mo5+6wg/QMEHFn/K5hYVJG+AuSiC9iMjQZd0v/uZSSK28PbyFwqWYHqro7BIOLwU',
> t: 'MTY4MzIyOTYyMC41MzUwMDA=',
> m: 'QI/Mz0Ee6U2tVKO7oyUm1rnTFY08iHsGZZgBWD//pJE=',
> i1: 'oGBcx46C2eJTRsnYoBl9Fw==',
> i2: 'ivG0C3fNJlZus2SYpxYgWg==',
> zh: 'bc2S/gxmZ7DjY83P/aPZxKt97roKTKr5gTZltM5CslI=',
> uh: 'SLdVolODg++SO356HusO5I/hbfOpiiOxQXj62i/MUkA=',
> hh: '64fwn+JmY1843Sj+i34nu291prlb33wsol1q87ekECc=',
> }
> };
> var trkjs = document.createElement('img');
> trkjs.setAttribute('src', '/cdn-cgi/images/trace/managed/js/transparent.gif?ray=7c2340c85dc337f2');
> trkjs.setAttribute('alt', '');
> trkjs.setAttribute('style', 'display: none');
> document.body.appendChild(trkjs);
> var cpo = document.createElement('script');
> cpo.src = '/cdn-cgi/challenge-platform/h/g/orchestrate/managed/v1?ray=7c2340c85dc337f2';
> window._cf_chl_opt.cOgUHash = location.hash === '' && location.href.indexOf('#') !== -1 ? '#' : location.hash;
> window._cf_chl_opt.cOgUQuery = location.search === '' && location.href.slice(0, location.href.length -
> window._cf_chl_opt.cOgUHash.length).indexOf('?') !== -1 ? '?' :
> location.search;
> if (window.history && window.history.replaceState) {
> var ogU = location.pathname + window._cf_chl_opt.cOgUQuery + window._cf_chl_opt.cOgUHash;
> history.replaceState(null, null, "\/en\/Magic\/Products\/Search?idCategory=0&idExpansion=0&searchString=%5Bsol+ring%5D&exactMatch=on&onlyAvailable=on&idRarity=0&sortBy=price_asc&perSite=1&__cf_chl_rt_tk=Y.l_DDIwIQE4RfQbth8EL91OZUHtyRmRaw_VKrnp_II-1683229620-0-gaNycGzNC-U" + window._cf_chl_opt.cOgUHash);
> cpo.onload = function() {
> history.replaceState(null, null, ogU);
> };
> }
> document.getElementsByTagName('head')[0].appendChild(cpo);
> }()); </script> </body> </html>
For the wanted HTML see cardmarket.com.
I have tried various things, like switching to other libraries like urllib3, in hopes that it is a requests specific problem, but the problem persists.
