Extract data behind authentication

Overview

This recipe demonstrates how to use Airtop to extract data from a website using a prompt. By leveraging Airtop’s live view capabilities, you can have your users log into any of their accounts inside a browser session to provide your agents access to content that requires authentication. Airtop profiles can be used to persist a user’s login state across sessions and avoid the need to have them log in again.

The instructions below will walk through creating a script that connects to Airtop, provides a live view for a user to log into their Glassdoor account if necessary, and retrieves a list of relevant job postings from the Glassdoor website. Similar logic can be applied to any website that requires authentication.

The full source code is available on GitHub for TypeScript and Python.

Prerequisites

To get started, ensure you have:

• An Airtop API key. You can get one for free.

and the following packages installed:

1
- Node.js installed on your system.

Getting Started

1. Clone the repository

   Start by cloning the source code from GitHub:

   NodeJSPython

   $
   git clone https://github.com/airtop-ai/recipe-prompt-content.git
   >
   cd recipe-prompt-content

2. Install dependencies

   Run the following command to install the necessary dependencies, including the Airtop SDK:

   NodeJSPython

   $
   npm install

3. Configure your environment

   You will need to provide your Airtop API key in a .env file. First, copy the provided example .env file:

   $
   cp .env.example .env

   Now edit the .env file to add your Airtop API key:

   AIRTOP_API_KEY=<YOUR_API_KEY>

Script Walkthrough

The script index.ts for TypeScript or extract_data_login.py for Python performs the following steps:

1. Initialize the Airtop Client

   First, we initialize the AirtopClient using your provided API key. This client will be used to create browser sessions and interact with the page content.

   NodeJSPython

   1
   const client = new AirtopClient({
   2
     apiKey: AIRTOP_API_KEY,
   3
   });

2. Create a Browser Session

   Creating a browser session will allow us to connect to and control a cloud-based browser. The API accepts an optional profileName parameter, which can be used to reuse a user’s previously provided credentials. If no profileName is given, the user will be prompted to log in at the provided live view URL (see Step 4). If the entered profile name does not exist, it will be created and saved on session termination for future use.

   NodeJSPython

   1
   const profileName: string | undefined = await new Promise<string | undefined>((resolve) => {
   2
     process.stdout.write('Enter a profile name(or press Enter to skip): ');
   3
     process.stdin.once('data', (input) => {
   4
       const trimmedInput = input.toString().trim();
   5
       resolve(trimmedInput || undefined);
   6
     });
   7
   });
   8
   9
   const createSessionResponse = await client.sessions.create({
   10
     configuration: {
   11
       timeoutMinutes: 10, // Terminate the session after 10 mins of inactivity, customize it as needed
   12
       profileName, // Profile for reusing or saving login credentials
   13
     },
   14
   });

3. Connect to the Browser

   The script opens a new page and navigates to the target URL, in this case Glassdoor’s user profile page.

   NodeJSPython

   1
   const windowResponse = await client.windows.create(
   2
     sessionId,
   3
     { url: TARGET_URL }, // TARGET_URL is defined at the top of the script
   4
   );

4. Handle Log-in Status

   If the user is not logged in, it waits for the user to log in at the provided live view URL.

   If the user is already logged in, they are navigated to the target URL to proceed with data extraction.

   NodeJSPython

   1
   const windowInfo = await client.windows.getWindowInfo(sessionId, windowResponse.data.windowId);
   2
   3
   // Querying the page content to determine if the user is logged in
   4
   const isLoggedInPromptResponse = await client.windows.pageQuery(session.id, windowInfo.data.windowId, {
   5
     prompt: IS_LOGGED_IN_PROMPT,
   6
   });
   7
   const parsedResponse = JSON.parse(isLoggedInPromptResponse.data.modelResponse);
   8
   if (parsedResponse.error) {
   9
     throw new Error(parsedResponse.error);
   10
   }
   11
   const isUserLoggedIn = parsedResponse.isLoggedIn;
   12
   13
   if (!isUserLoggedIn) {
   14
     console.log(
   15
       'Log into your Glassdoor account on the live view of your browser window.  Press `Enter` once you have logged in.',
   16
       windowInfo.data.liveViewUrl,
   17
     );
   18
     await new Promise<void>((resolve) => process.stdin.once('data', () => resolve())); // Wait for the user to log in
   19
   } else {
   20
     console.log(
   21
       'User is already logged in. View progress at the following live view URL:',
   22
       windowInfo.data.liveViewUrl,
   23
     );
   24
   }

5. Navigate to the Target URL

   After logging in, the script navigates to the target URL, which in this case is a Glassdoor search page for software engineering jobs in San Francisco.

   NodeJSPython

   1
    await page.goto(TARGET_URL);

6. Query the AI to Extract Data

   We construct a prompt that asks the AI to extract data about job postings that are related to AI companies. We also define a JSON schema for the output. Note that an optimal prompt will begin by providing context about the webpage and what the model is viewing. It will also include information in the description fields of a provided JSON schema to guide the model’s output.

   NodeJSPython

   1
   const PROMPT = `This browser is open to a page that lists available job roles for software engineers in San Francisco. Please provide 10 job roles that appear to be posted by the AI-related companies.`;
   2
   3
   const SCHEMA = {
   4
     $schema: 'http://json-schema.org/draft-07/schema#',
   5
     type: 'object',
   6
     properties: {
   7
       companies: {
   8
         type: 'array',
   9
         items: {
   10
           type: 'object',
   11
           properties: {
   12
             companyName: {
   13
               type: 'string',
   14
             },
   15
             jobTitle: {
   16
               type: 'string',
   17
             },
   18
             location: {
   19
               type: 'string',
   20
             },
   21
             salary: {
   22
               type: 'object',
   23
               properties: {
   24
                 min: {
   25
                   type: 'number',
   26
                   minimum: 0,
   27
                 },
   28
                 max: {
   29
                   type: 'number',
   30
                   minimum: 0,
   31
                 },
   32
               },
   33
               required: ['min', 'max'],
   34
             },
   35
           },
   36
           required: ['companyName', 'jobTitle', 'location', 'salary'],
   37
         },
   38
       },
   39
       error: {
   40
         type: 'string',
   41
         description: 'If you cannot fulfill the request, use this field to report the problem.',
   42
       },
   43
     },
   44
   };

   Utilizing Airtop’s prompt feature, the script requests data about job postings that are related to AI companies, formatted as per the provided JSON schema. The AI agent can follow pagination links to gather more results on sites with multiple pages or from a feed with infinite scrolling.

   NodeJSPython

   1
   // Note that it may take several minutes to receive a response from the AI agent
   2
   // as it follows pagination links and gathers data from each page
   3
   const response = await client.windows.pageQuery(session.id, windowInfo.data.windowId, {
   4
     prompt: PROMPT,
   5
     followPaginationLinks: true,
   6
     configuration: {
   7
       outputSchema: SCHEMA,
   8
     },
   9
   });
   10
   11
   // Print the result to the console or otherwise use it as desired
   12
   console.log(response.data.modelResponse);

7. Clean Up

   Finally, the script closes the browser and terminates the session.

   NodeJSPython

   1
   await browser.close();
   2
   await client.sessions.terminate(session.id);

Running the Script

To run the script, execute the following command in your terminal:

$
npm run start

Summary

This recipe showcases how Airtop can be used to automate tasks that require authentication and data extraction from dynamic content. By combining Airtop’s live view feature for manual login with automated data extraction via natural language prompts, you can interact with and extract data from complex websites that require user credentials.