DEV Community

Cover image for Advanced Automation Tips with Python | Selenium
Rashid
Rashid

Posted on • Updated on

Advanced Automation Tips with Python | Selenium

What's up DEV Network?

In this post I want to share my automation experience from my current internship.

Youtube Channel with video tutorials - Reverse Python Youtube

But before we dive in please visit Reverse Python if you want to find more articles like this.

Actually, I wanted to explain it to you on a real project, but unfortunately, I didn't find any platform that fits all my requirements.

However, if you are Python developer you will need these tips to build your automation program fast. So, let's start!

Web Driver

It is important to configure web driver correctly to be able to run automation. If you want to use Chrome as a web driver then you should install chromedriver. However, if you want to choose Firefox then you should install geckodriver.

Creating a class

With creating a class you can easily handle URLs and call functions so you don't have to create web drivers in each function. Take a look following code:

from selenium import webdriver

class Bot:

    def __init__(self):
        self.bot = webdriver.Firefox(executable_path='/path/to/geckodriver')

automate = Bot() 
Enter fullscreen mode Exit fullscreen mode

So, when you need driver in function just write:

  def search(self):
        bot = self.bot
        bot.get('www.google.com')    
Enter fullscreen mode Exit fullscreen mode

Handle Login

For handle login you should add fields into __init__ function. Take a look following code:

import time
from selenium import webdriver
from selenium.webdriver.common.keys import Keys

class Bot:

    def __init__(self,username,password):
        self.username = username
        self.password = password
        self.bot = webdriver.Firefox(executable_path='/path/to/geckodriver')

    def login(self):
        bot = self.bot
        bot.get('https://website.com/')
        time.sleep(3)
        email = bot.find_element_by_id('userNameBox')
        password = bot.find_element_by_id('passWordBox')
        email.clear()
        password.clear()
        email.send_keys(self.username)
        password.send_keys(self.password)
        bot.find_element_by_id('LoginBtn').click()
        time.sleep(3) 

automate = Bot('your_username', 'your_password') 
automate.login()
Enter fullscreen mode Exit fullscreen mode

Very simple approach to handle logins.

Looping clicks

It is important to use time.sleep() if you are looping clicks. Because automation needs time to perform another click. If you don't use time.sleep() then ElementClickInterceptedException error will appear.

for elem in elements:
        elem.click()
        time.sleep(3)
Enter fullscreen mode Exit fullscreen mode

Write data into CSV

There is a lot of solutions and examples on the internet about how to write data to CSV. Let's assume that you have multiple arrays and you want to write them into CSV under the right field names (column name).

Let me be more clear..

My task was to crawl some phrases and translate them to all languages then write all these data into CSV. So, I used GT API and created an array for each language.

I had a lot of arrays so I need somehow write each translation under right field names (column name).

The following code demonstrating the best solution to handle these kinds of problems:

# zip arrays
languages_translations = zip(ar,hy,ms,bg,zh_cn_SAR,zh_cn,
zh_tw_singapore,zh_tw,hr,cs,da,nl,en_australia,en_uk,en_usa,et,fi)    

# Write to CSV
with open('translations.csv', mode='w', newline='', encoding="utf-8") as csv_file:
    fieldnames = ['Arabic', 'Armenian (Armenia)', 'Bahasa Malaysia (Malaysia)', 'Bulgarian (Bulgaria)', 'Chinese (Hong Kong SAR)', 
    'Chinese (Simplified)','Chinese (Singapore)','Chinese (Traditional)','Croatian (Croatia)', 'Czech (Czech Republic)',
    'Danish (Denmark)','Dutch (The Netherlands)','English (Australia)','English (UK)', 'English (US)','Estonian (Estonia)','Finnish (Finland)']
    writer = csv.DictWriter(csv_file, fieldnames=fieldnames)
    writer.writeheader()
    for lang in languages_translations:
        writer.writerow(dict(zip(fieldnames, lang)))  
Enter fullscreen mode Exit fullscreen mode

As you see, you should zip all arrays, loop it and write rows dictionary of zipped field names and loop items.

By this way you will write all data correctly into CSV file.

Handle Popups or Iframes

Another challenging task is to handle popups or iframes while automation. If you need to interact with popup or iframe elements, you should tell selenium to switch the window from main to popup. Once you finished with popup you should switch back to the main window.


# find iframe or popup element
iframe = bot.find_element_by_tag_name('iframe')
# switch to iframe or popup window
bot.switch_to.frame(iframe)
# interact with elements
textarea = bot.find_element_by_tag_name('body')
textarea.send_keys('Some Keys Here')
# switch back to main window
bot.switch_to.default_content() 
Enter fullscreen mode Exit fullscreen mode

Great! I am providing you the solutions which is really complicated on internet.

If you have to handle alert popups you also can use this method.

Send CSV data to web elements

Sometimes you need to send csv data into web elements such as multiple textboxes.

Assume that you have a multiple panels with multiple textboxes in it and you have to send CSV rows into these texboxes

textboxes

So each CSV row belongs different panels. Sounds crazy right? This is how you will train your brain :) Somehow you have to iterate each CSV row for each panel's textboxes.

Before we go on please visit Reverse Python if you want to find more articles like this.

Alrgiht! Let's see the code and then I am going to explain:

def send_keys_textboxes(self,url):
    bot = self.bot
    # go to url
    bot.get(url)
    # reader object which will iterate over lines in the given csvfile
    with open('translations.csv', 'rt', encoding='utf8') as csvfile:
        langs = csv.reader(csvfile, delimiter=',', quotechar='"')
        # create list of langs
        langs = list(langs)
    elements = bot.find_elements_by_xpath("//a[@data-tag='globalize']")
    # Using index to handle multiple panels
    index = 0
    try:
        for elem in elements:
            class_of_element = elem.get_attribute("class")
            if class_of_element == 'cs-trans-icon':
                # Panel opens with click
                elem.click()
                time.sleep(3)
                textBoxes = bot.find_elements_by_tag_name('textarea')
                # Loop only specific index of list 
                phrases = langs[index]
                # Itearating TextBoxes
                for i in range(len(phrases)):
                    textBoxes[i].send_keys(phrases[i].title())
                time.sleep(3)
                # Increasing index for next panel
                index = index+1
                try:
                    bot.find_element_by_class_name('CsImageButton').click()
                except NoSuchElementException:
                    bot.find_element_by_class_name('cso-btn').click()
            time.sleep(3)  
    except ElementClickInterceptedException:
        pass 
Enter fullscreen mode Exit fullscreen mode

Index plays main role in this solution. We said that we want to handle multiple panels, so index prevents to loop all rows in each panel. Assume that you have 3 rows and without using index, program will continue to loop until 3rd row finished. As a result you will have 3 values for each textbox. However, we need to send only first row values into first panel's textboxes, second row values into second panel's textboxes and so on..

Once index is defined, we are iterating textboxes. By this way, first value in row is going to send first textbox, second value in row is going to send second textbox and so on..

These examples are from the real-world project so I recommend to bookmark this post.

Handle Dropdowns

So there are 2 approaches for handling dropdwons.

First solution is to use selenium's Select method to select options from dropdown.

from selenium.webdriver.support.ui import Select

bot = self.bot
bot.get("https://example.com")
select = Select(bot.find_element_by_xpath("//select"))
# select bu index of option
select.select_by_index(2)
# select by text of option
select.select_by_visible_text('Visible Text')
# select by value of option
select.select_by_value('value')
Enter fullscreen mode Exit fullscreen mode

and second solution is just using xpath to click dropdown items:

bot.find_element_by_xpath("//select/option[text()='Option_Text_Here']").click()
Enter fullscreen mode Exit fullscreen mode

Navigate parent element with xpath

Sometimes Html structures can be messed up and you can't select the element you want. In these cases, it is good to navigate the parent element or go back using xpath:

# elem is a web element object
# selenium will jump 2 step back to look for parent element
 parent_element = elem.find_element_by_xpath('..').find_element_by_xpath('..')
Enter fullscreen mode Exit fullscreen mode

Mission Accomplished!

That is all for now! I shared my python automation experience with you and I hope it helps you and saves your time from searching on the internet.

Please visit Reverse Python for more articles (you won't disappointed) and let me know in comments what my next post should be about.

See you very soon DEVs! Stay Connected!

Top comments (7)

Collapse
 
individualit profile image
Artur Neumann

"It is important to use time.sleep() if you are looping clicks. Because automation needs at least 2 seconds to perform another click. If you don't use time.sleep() then ElementClickInterceptedException error will appear."

In my experience there is no way to tell how long it will take to click and specially get a result of some sort. It depends on the app, on the computer the test runs on, on the browser and on the constellation of planets in the solar system.

Those sleeps will make your tests flaky (pass sometimes and break in other times), specially when running in CI on unknown hardware
same for stuff like:

bot.get('https://website.com/')
time.sleep(3)

You will better off to wait for the result of your action. Some waiting indicator disappearing, some specific field to appear that indicates that the side is loaded. Basically make a loop and wait for some event that would also indicate to the normal user that the site is ready for the next action

Collapse
 
dkondraszuk profile image
dkondraszuk • Edited
def search(self):
        bot = self.bot
        bot.get('www.google.com')    

Why would you ever do that. self.bot.get... is just fine, no need to copy attributes.

bot.find_element_by_id('LoginBtn').click()
time.sleep(3)

If you are performing another operation after logging then time.sleep is not a good practice. For this you have better approach like implicit explicit waits.

Collapse
 
thedevtimeline profile image
Rashid

The objective in this post is to be more clear for those who are starting automation. So, in my view self.bot.get() not looks clean and organized. If we talk about waits, it causes different kinds of errors depending on website's speed. Even sometimes websites can cause internal errors so these waits will crash at that moment. Anyway, thank you for your comment maybe your approach will work for someone👍

Collapse
 
hyftar profile image
Simon Landry

Before reading this comment, please visit Reverse Python for more articles like this.

God are you this desperate for visitors?

Collapse
 
programingtube profile image
Big Box

Is there any alternative to selenium for python. Coz everytime the Chrome Updates selenium requires a compatible Web Driver and after distributing my application it becomes owing to users.

Collapse
 
sh99 profile image
Shane Hu

Hi big box, try webdriver_manager

Collapse
 
williamhruska11 profile image
William Hruska

Great post.