Google Lens APIs/SDK

Question

I was going through the Google Cloud Vision API to search for related products in the image. However, I found that I need to first create a product set (and upload the images) and then search on the set that I've created.

Is there any API via which I can provide an image (without creating any product set) and search google (or any specific website like amazon) for the related products (and their eCommerce links) in the image? Basically replicating what google lens does for its app. If not, is there any google lens mobile SDK for the same use-case, ie finding the related products (and their eCommerce URI) provided an image??

Update 1

I researched a bit more on this, and found Web Detection API. This does serve the purpose somewhat. However, I cant filter the results from a specific website (for eg:- Amazon). Is there a way to filter results from a specific domain?

Oh yes, it did. However, we opted out of this approach bcz of pricing issues. Nonetheless, thank you so much for the same. — Praful Bagai, May 13 '22 at 05:09

score 3 · Accepted Answer · edited Feb 02 '23 at 11:37

With Vision API, you can detect from the web properties, faces, and text in the images.

An option is to use filtering to return more specific results when listing resources, evaluations, or operations. Using ?filter="[filter_name=]filter-value returns all JSON objects that match the filter-value. You can see more information in this link.

Another option is to filter before showing the results. You can see this example, which returns links from the Web and cloud storage that contain certain colors in the images.

You can see this example:

def detect_properties_uri(uri):
    """Detects image properties in the file located in Google Cloud Storage or
    on the Web."""
    from google.cloud import vision
    client = vision.ImageAnnotatorClient()
    image = vision.Image()
    image.source.image_uri = uri
 
    response = client.image_properties(image=image)
    props = response.image_properties_annotation
    print('Properties:')
 
    for color in props.dominant_colors.colors:
        print('frac: {}'.format(color.pixel_fraction))
        print('\tr: {}'.format(color.color.red))
        print('\tg: {}'.format(color.color.green))
        print('\tb: {}'.format(color.color.blue))
        print('\ta: {}'.format(color.color.alpha))
 
    if response.error.message:
        raise Exception(
            '{}\nFor more info on error messages, check: '
            'https://cloud.google.com/apis/design/errors'.format(
                response.error.message))
# [END vision_image_property_detection_gcs]

The next step is to iterate through the results like this example and filter the links that contain the domain. Looking at the domain inside the web links.

for entity in web_detection.web_entities:
    if (str(entity.description).find(domain) >-1)
        print(entity.description)

In this case, you will iterate each result and filter trying to find the domain inside the description and display it on the screen.

Another option would be to manipulate the result in JSON format. In that chase, you need to:

Parse JSON
Filter the JSON

You can see this example:

import json
 
input_json = """
[
    {
        "type": "1",
        "name": "name 1"
    },
    {
        "type": "2",
        "name": "name 2"
    },
    {
        "type": "1",
        "name": "name 3"
    }
]"""
 
# Transform json input to python objects
input_dict = json.loads(input_json)
 
# Filter python objects with list comprehensions
output_dict = [x for x in input_dict if x['type'] == '1']
 
# Transform python object back into json
output_json = json.dumps(output_dict)
 
# Show json
print output_json

score 1 · Answer 2 · edited Apr 12 '23 at 06:33

Make a GET request to https://lens.google.com/uploadbyurl?url=YOUR_IMAGE_URL

You will also need to play around with your headers. I think User-Agent is necessary along others.

Then parse the html response.

(Ik she ain't pretty but she runs, let me know how can I can improve it)

import json, bs4, request

def extract_data(soup):
    """This function is used to extract the data of a google lens page (response of bd_proxy_search  )

    Args:
        soup (bs4.BeautifulSoup): the html of the page (response of bd_proxy_search  )
    Returns:
        product_list (list): a list of dictionaries containing the data"""
    # Finds an element containig 'Visual Matches'  and returns a json object containing the data
    script_elements = soup.find_all('script')

    raw_data = [x for x in script_elements if 'Visual matches' in x.text]

    raw_data = raw_data[0].text

    start = raw_data.find('data:')+5
    end = raw_data.find('sideChannel') -2
    json_data = json.loads(raw_data[start:end])

    jason = []

    ###########################################
    # This is used beacuse sometimes the information is in json_data[1][0] and other times in json_data[1][1]
    try:
        jason = json_data[1][1][1][8][8][0][12] if len(json_data[1]) == 2 else json_data[1][0][1][8][8][0][12]
    except:
        print("The data is not in the expected format")
    ###########################################

    product_list = []
    for product in jason:
        information = {
            'google_image': product[0][0],
            'title': product[3],
            'redirect_url': product[5],
            'redirect_name': product[14],
            # 'price': product[0][7][1] if len(product[0]) > 6 else None

        }
        product_list.append(information)

    return product_list

This should return a list of dictionaries containing the information you need.

GL

This is an amazing hack. However, it does not display the related shopping products that the person is wearing in the image. IG, its only a functionality within the mobile app — Praful Bagai, Apr 20 '23 at 16:01
You should be getting back a dictionary containing info about related products. Could you describe your issue in further detail ? — Alvpvtos, Jun 16 '23 at 14:07

Google Lens APIs/SDK

2 Answers2