How can I make a search term in Apify a variable using Google App Script

Question

I am trying to change the search query in the Apify Google Search Scraper using Google App Script by making the search term a variable. https://apify.com/apify/google-search-scraper

I am trying to see if i can reference it by its code. https://github.com/apifytech/actor-google-search-scraper/blob/master/src/main.js

I am getting this

<!DOCTYPE html>
<html>
<head>
  <link rel="stylesheet" type="text/css" class="__meteor-css__" href="https://cdn.apify.com/35dedb64425e459215b6d9b580601d18eccb1d29.css?meteor_css_resource=true">
<meta charset="utf-8" />
    <title>My Apify</title>
    <meta name="description" content="Apify (formerly Apifier) is the world’s most advanced web scraping and automation platform. Turn any website into an API in a few minutes." />
    <meta name="copyright" content="Copyright&copy; 2018 Apify Technologies s.r.o. All rights reserved." />
    <meta name="keywords" content="web scraper, web crawler, data extraction, API" />
    <meta name="robots" content="index, follow" />
    <meta name="referrer" content="origin" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />

    <link href="https://fonts.googleapis.com/icon?family=Material+Icons" rel="stylesheet" />

    <meta property="og:title" content="Apify - The web scraping and automation platform" />
    <meta property="og:site_name" content="Apify" />
    <meta property="og:url" content="https://apify.com" />
    <meta property="og:type" content="website" />
    <meta property="og:description" content="Apify (formerly Apifier) is the world’s most advanced web scraping and automation platform. Turn any website into an API in a few minutes." />
    <meta property="fb:app_id" content="1636933253245869" />
    <meta property="og:image" content="https://apify.com/img/og-image.png" />
    <meta property="og:image:width" content="1200" />
    <meta property="og:image:height" content="630" />
    <meta property="og:image" content="https://apify.com/img/og-image-square.png?v=2">
    <meta property="og:image:width" content="1000" />
    <meta property="og:image:height" content="1000" />

    <link rel="icon" type="image/svg+xml" href="https://apify.com/img/favicon.svg" sizes="any" />
    <link rel="icon" type="image/png" href="https://apify.com/img/favicon-128x128.png" sizes="128x128" />
    <link rel="icon" type="image/png" href="https://apify.com/img/favicon-96x96.png" sizes="96x96" />
    <link rel="icon" type="image/png" href="https://apify.com/img/favicon-64x64.png" sizes="64x64" />
    <link rel="icon" type="image/png" href="https://apify.com/img/favicon-48x48.png" sizes="48x48" />
    <link rel="icon" type="image/png" href="https://apify.com/img/favicon-32x32.png" sizes="32x32" />
    <link rel="icon" type="image/png" href="https://apify.com/img/favicon-16x16.png" sizes="16x16" />
    <link rel="icon" type="image/x-icon" href="https://apify.com/favicon.ico" sizes="16x16 32x32 48x48 64x64" />
    <link rel="apple-touch-icon" href="https://apify.com/img/apple-touch-icon-180x180.png" sizes="180x180" />
    <script type="text/javascript" defer id="ie-check-script">
        var userAgent = navigator.userAgent.toUpperCase();
        // IE 11 uses user agent containing Trident as browser identifier, while IE bellow 11 has MSIE as browser identifier
        var browserIsIE = userAgent.indexOf('TRIDENT/') != -1 || userAgent.indexOf('MSIE') != -1;
        // If browser is detected as IE we show warning instead of document body, otherwise we remove this script
        if (browserIsIE) {
            var chromeLink = '<a href="https://www.google.com/chrome/">Google Chrome</a>';
            var firefoxLink = '<a href="https://www.mozilla.org/cs/firefox/new/">Mozilla Firefox</a>';
            var edgeLink = '<a href="https://www.microsoft.com/en-us/windows/microsoft-edge">Microsoft Edge</a>';
            var safariLink = '<a href="https://www.apple.com/safari/">Apple Safari</a>';
            var warningStyles = 'display: inline-block; margin: 10px; padding: 25px; border: 1px solid rgb(255, 200, 200); border-radius: 4px; font-size: 16px; background-color: rgb(255, 240, 240);';

            var message = '<div style="text-align:center;"><div style="'+warningStyles+'">We are sorry, but your browser is currently not supported by our platform.<br />'
                +'Please download one of the supported browsers: '+chromeLink+', '+firefoxLink+', '+edgeLink+' or '+safariLink+'.</div></div>';
            var newBody = document.createElement('body');
            newBody.innerHTML = message;
            document.body = newBody;
        } else {
            var thisScript = document.querySelector('#ie-check-script');
            thisScript.parentElement.removeChild(thisScript);
        }
    </script>

</head>
<body>
  <script type="text/javascript">__meteor_runtime_config__ = JSON.parse(decodeURIComponent("%7B%22meteorRelease%22%3A%22METEOR%401.10.1%22%2C%22gitCommitHash%22%3A%223a3adab260deebeece444a11c4a79a49bd6b60e5%22%2C%22meteorEnv%22%3A%7B%22NODE_ENV%22%3A%22production%22%2C%22TEST_METADATA%22%3A%22%7B%7D%22%7D%2C%22PUBLIC_SETTINGS%22%3A%7B%22sentryDsn%22%3A%22https%3A%2F%2Fb61d4f03051b49e4ac4162e9e14314e8%40sentry.io%2F1494973%22%2C%22sentryEnv%22%3A%22prod%22%2C%22webScraperActorId%22%3A%22moJRLRc85AitArpNN%22%2C%22puppeteerScraperActorId%22%3A%22YJCnS9qogi9XxDgLB%22%2C%22legacyPhantomjsCrawlerActorId%22%3A%22YPh5JENjSSR6vBf2E%22%2C%22analyticsSettings%22%3A%7B%22Google%20Analytics%22%3A%7B%22trackingId%22%3A%22UA-67003981-5%22%7D%2C%22GoogleTagManager%22%3A%22GTM-MNGXGGB%22%2C%22OLD-Mixpanel%22%3A%7B%22token%22%3A%22b7bc52869761789650547589c1c97344%22%2C%22people%22%3Atrue%7D%7D%2C%22intercom%22%3A%7B%22id%22%3A%22kod1r788%22%2C%22allowAnonymous%22%3Atrue%7D%2C%22rewriteApiUrl%22%3A%7B%22from%22%3A%22https%3A%2F%2Fmy.apify.com%2Fapi%2F%22%2C%22to%22%3A%22https%3A%2F%2Fapi.apify.com%2F%22%7D%2C%22apiServerUrl%22%3A%22https%3A%2F%2Fapi.apify.com%22%2C%22cdnPrefix%22%3A%22https%3A%2F%2Fcdn.apify.com%22%2C%22staticWebUrl%22%3A%22https%3A%2F%2Fapify.com%22%2C%22docsUrl%22%3A%22https%3A%2F%2Fdocs.apify.com%22%2C%22cookieDomain%22%3A%22apify.com%22%2C%22cookieNames%22%3A%7B%22userId%22%3A%22ApifyProdUserId%22%2C%22user%22%3A%22ApifyProdUser%22%7D%2C%22superProxy%22%3A%7B%22hostname%22%3A%22proxy.apify.com%22%2C%22port%22%3A%228000%22%2C%22statusPageUrl%22%3A%22http%3A%2F%2Fproxy.apify.com%22%2C%22googleSerpProxyGroupId%22%3A%22GOOGLESERP%22%7D%2C%22conductor%22%3A%7B%22hostname%22%3A%22runs.apify.net%22%2C%22port%22%3A80%7D%2C%22appBanner%22%3Anull%2C%22userUploadedFilesS3BucketName%22%3A%22apify-uploads-prod%22%2C%22theaterServerMemoryMbytes%22%3A65536%2C%22imageProxyOptions%22%3A%7B%22hmacKey%22%3A%22BUsn3bQLa6wtN9iyQs64at3b%22%2C%22domain%22%3A%22apifyusercontent.com%22%7D%2C%22mixpanelToken%22%3A%226706f12b3a8942be188202d48458aa5b%22%2C%22enableClientDebugLogs%22%3Afalse%7D%2C%22ROOT_URL%22%3A%22https%3A%2F%2Fmy.apify.com%22%2C%22ROOT_URL_PATH_PREFIX%22%3A%22%22%2C%22accountsConfigCalled%22%3Atrue%2C%22autoupdate%22%3A%7B%22versions%22%3A%7B%22web.browser%22%3A%7B%22version%22%3A%2245d4d39e1c0c5679475d46bdb28c75d4b662941f%22%2C%22versionRefreshable%22%3A%2257063c4cf996485c3d1c14b0fc943ec069c52a98%22%2C%22versionNonRefreshable%22%3A%22053bcfd91a068917fd9fbe891cd9c3feb0f0bffb%22%7D%2C%22web.browser.legacy%22%3A%7B%22version%22%3A%223757fa1060d2a68657832aac83619ffa2113a97b%22%2C%22versionRefreshable%22%3A%2257063c4cf996485c3d1c14b0fc943ec069c52a98%22%2C%22versionNonRefreshable%22%3A%22897380d0290904c3f33ff3960c3cd9ae92a38a0c%22%7D%7D%2C%22autoupdateVersion%22%3Anull%2C%22autoupdateVersionRefreshable%22%3Anull%2C%22autoupdateVersionCordova%22%3Anull%2C%22appId%22%3A%225v8kbc7bvh2d1uwv25j%22%7D%2C%22appId%22%3A%225v8kbc7bvh2d1uwv25j%22%2C%22isModern%22%3Afalse%7D"))</script>

  <script type="text/javascript" src="https://cdn.apify.com/280c18e7fd3d10ff3f09510877315f175acff7a7.js?meteor_js_resource=true"></script>


</body>
</html>

My code

function runSearch() {
  var formData = {
    initialRequests : 'banks in phoenix',
  };
  var options = {
    'method' : 'post',
    'payload' : formData
  };
  const test = UrlFetchApp.fetch('https://my.apify.com/tasks/<my-task-id>', options);
  Logger.log(test);
}

What do I need to change?

score 2 · Answer 1 · answered Apr 01 '20 at 21:17

There are many things that you need to change.

1) You are using your private logged-in URL - https://my.apify.com/tasks/xLojrg7GEYkQbGBBL. Instead, you need to check the API tab on the correct URL to call this task via API - https://api.apify.com/v2/actor-tasks/xLojrg7GEYkQbGBBL/runs?token=YOUR_TOKEN (swap in your Apify API token from your account tab)

2) The field where you pass your queries is not called initialRequests but queries. Check out your task, switch your input to JSON and you will see how the fields are called in JSON.

3) Unfortunately, right now Apify doesn't allow getting dataset data (like from Google Search Scraper) via single API call for longer runs. Check out this article that explains how to integrate via API.

Let Me Tink About It · Answer 2 · 2020-04-01T23:11:36.800

Get content text of HTTPResponse.

Change:

Logger.log(test);

to:

Logger.log(test.getContentText());

Per the UrlFetchApp docs, the fetch() method returns an object of type: HTTPResponse. You want to get the content text by applying the getContentText() method to the HTTPResponse object. As follows.

// Make a GET request and log the returned content.
var response = UrlFetchApp.fetch('http://www.google.com/');
Logger.log(response.getContentText());

Change request parameter to conform to documentation.

Change:

initialRequests : 'banks in phoenix',

to:

queries: 'banks in phoenix',

Look here at the Google Search Results Scraper documentation.

The following table shows specification of the actor INPUT fields as defined by its input schema. These fields can be [...] provided in a JSON object when running the actor using the API. Read more in docs.

...

Search queries or URLs

Google Search queries (e.g. food in NYC) and/or full URLs (e.g. https://www.google.com/search?q=food+NYC).

Enter one item per line.

Optional
Type: String
JSON example
"queries": "Hotels in NYC
  Restaurants in NYC
  https://www.google.com/search?q=restaurants+in+NYC"

So, to summarize, try changing your code to the following.

const runSearch = () => {
  const formData = {
    queries: 'banks in phoenix',
  };
  const options = {
    method: 'post',
    payload: formData,
  };
  const test = UrlFetchApp.fetch(
    'https://my.apify.com/tasks/<my-task-id>',
    options,
  );
  const response = test.getContentText();
  Logger.log( 'response\n%s', response, );
}

'https://my.apify.com/tasks/' is a wrong URL. That goes to the app. He or she needs to use the API URL.- https://api.apify.com/v2/actor-tasks/xLojrg7GEYkQbGBBL/runs?token=YOUR_TOKEN — Lukáš Křivka, Apr 02 '20 at 22:00

How can I make a search term in Apify a variable using Google App Script

2 Answers2