How to use Apify to log in to a site and click a button?

Question

I need use Apify and Zapier to automate i) logging-in to a password-protected web page and ii) clicking a button. How can I do this?

I think I should be using Puppeteer in Actor, but I'm not certain how.

Target URLs will change from time to time. Their format is https://studio.example.com/products/videocloud/media/videos/{id_code} where {id_code} is, for example, 6091481925001.

1. Zapier

Zapier should invoke an Apify Actor to do the work. An action in an existing zap already has access to the dynamic {id_code}. An additional action should "Run Actor" to Apify, passing either {id_code} or full URL https://studio.example.com/products/videocloud/media/videos/6091481925001 to Apify to run on.

How do I properly pass the value to Apify through "Input Body"?

2. Login

When accessed whilst unauthenticated, the page redirects to a login form at https://signin.example.com/?redirect=https%3A%2F%2Fstudio.example.com%2Fproducts%2Fvideocloud%2Fmedia%2Fvideos%2F6091481925001 with:

• "Email Address" (input with id="email" and name="email")
• "Password" (input with id="password" and name="password")
• "Sign In" button (button with id="signinButton" and type="submit")

How do I use an Actor to log in here?

3. Click

Once authenticated, the target page appears. It has a button bar including the button "Activate" (button whose child span text must only be "Activate" if we are allowed to click it).

(For info - once clicked, the button text should become "Deactivate").

How do I get Apify to click the "Activate" button here?

As I understand it, this is not a scraping job, since I am not looking to return data from the web page, so I should not be using apify/puppeteer-scraper or apify/web-scraper.

Update:

So far, I have the following. However, Puppeteer inside Apify times out - at fairly lengthy values up to 9000ms, suggesting it isn't necessarily a page-load issue (?)

const Apify = require('apify');

    Apify.main(async () => {



        // Get credentials
        const { EMAIL, PASSWORD } = Apify.getEnv();         // Docs for using values: https://apify.com/docs/actor#source-env-vars


        // Launch Puppeteer
        const browser = await Apify.launchPuppeteer();
        const page = await browser.newPage();
        await page.goto('https://signin.example.com/login');

        // Login
        await page.type('#email', process.env.EMAIL);
        console.log('Attempted to enter email');

        await page.type('#password', process.env.PASSWORD);
        console.log('Attempted to enter password');

        await page.click('#signinButton');
        console.log('Attempted to click button');    

        // Times-out here

        await page.waitForNavigation();
        console.log('Attempted to wait for navigation');

        // Get cookies
        const cookies = await page.cookies();
        console.log('Attempted to wait for cookies');

        await browser.close();

        console.log('Done.');


    });

Answer 1

1) You can pass arbitrary JSON to the input. It makes sense to pass just the as

{
   "id_code": "ID_CODE_FROM_ZAPIER"
}

2) On Apify's side, you will need to read the input first with

const input = await Apify.getInput();
const { id_code } = input;

Then you need to get your credentials. If they don't change, I would save them as the environment variables of the actor. If you name them EMAIL and PASSWORD, you can then access them in the code via

const { EMAIL, PASSWORD } = Apify.getEnv();

Now you need to launch Puppeteer, go to the login page, fill the input fields and click submit. It is pretty easy, the examples are shown in this article. You don't need to worry about the cookies now.

3) Once you log in, you need to go to your desired URL via

await page.goto(`https://studio.example.com/products/videocloud/media/videos/${id_code}`)

To find the exact element to click when it doesn't have useful selectors is doable with JQuery and finding the text. I cannot log in so I'm not 100% sure this will work.

// We need to inject JQuery first
await Apify.utils.puppeteer.injectJQuery(page);

// We can use JQuery only in the browser context, which means inside evaluate
await page.evaluate(() => {
    $('button:contains("Activate")').click()
})

Answer 2

1. You could play with the waitUntil parameter e.g.

await page.waitForNavigation({
   waitUntil: "networkidle2"  
});

** Check the docs for more info https://pptr.dev/#?product=Puppeteer&version=v1.20.0&show=api-pagewaitfornavigationoptions

2. Or you could wait for an element selector to appear in page e.g.

await page.waitForSelector('.buttonSelector')

** Check the docs for more info https://pptr.dev/#?product=Puppeteer&version=v1.20.0&show=api-pagewaitforselectorselector-options