0

I am trying to use Google Apps Script to scrape the first ten results of this page: https://www.tenderboard.gov.bh/Tenders/Opened%20Bids/.

Currently, I have written the POST request to the website. However, the response is coming back in a format that I am unsure how to process. The code I have written so far looks like this:

function fetchData() {
  var url = "https://www.tenderboard.gov.bh/Templates/TenderBoardWebService.aspx/GetOpenedTenderF";
  var formData = {
    "tenderNumber":"",
    "ministry":"0",
    "category":"0",
    "openedDate_filter":"",
    "sortingType":"0",
    "pageIndex":"2"
    };
  var payload = JSON.stringify(formData);

  var params = {
    "method":"post",
    "contentType":"application/json; charset=UTF-8",
    "payload":payload
  };
  
  var response = UrlFetchApp.fetch(url,params);

To illustrate its format, a sample of the response is:

{"d":"\u003cNewDataSet\u003e\r\n  \u003cPager\u003e\r\n    \u003cPageIndex\u003e1\u003c/PageIndex\u003e\r\n    \u003cPageSize\u003e10\u003c/PageSize\u003e\r\n    \u003cRecordCount\u003e3799\u003c/RecordCount\u003e\r\n    \u003cContent\u003e\u0026lt;div class=\u0027table-head\u0027 style=\u0027display: table-header-group; opacity: 1;\u0027\u0026gt;\u0026lt;div class=\"column\" data-label=\"No.\"\u0026gt;No.\u0026lt;/div\u0026gt;\u0026lt;div class=\"column\"

How would I go about extracting the contents of this response to build a 2D array containing the same information as the web page?

Many thanks and apologies for what I am sure is a dense question.

user16481624
  • 1
  • It's very easy to scrape using scrapy with pagination as data is sending in backdoor with post method. – Md. Fazlul Hoque Jul 19 '21 at 16:25
  • 1
    The https://www.tenderboard.gov.bh data looks to be controlled by a script. Upon double checking the complete response data, it contains no data similar information as to the actual web page information to be scraped out. So it looks it is not possible to scrape data from the response that's the same as to the information on the web page using apps script. Even though this is about IMPORTXML function, you may view the explanation from an answer at https://stackoverflow.com/a/65667104/15384825 about scraping website data that are being controlled by a script as it's also a very similar process. – SputnikDrunk2 Jul 19 '21 at 18:17

0 Answers0