Advertisement
  1. Code
  2. Python

PyQuery: Python's jQuery

Scroll to top

In this tutorial, you'll have a look at PyQuery, a Python library which allows you to make jQuery queries on XML documents. Syntactically it's quite similar to jQuery, and if you are familiar with jQuery it should be easier to follow.

Getting Started With PyQuery

To get started with PyQuery, install the Python package using PIP.

1
pip install pyquery
 

Once you have installed PyQuery, import it into the Python program.

1
from pyquery import PyQuery as pq

Once you have imported the PyQuery class, you can use it to load data from a string, a file, or even a URL.

Let's start with a basic example to see how it works. Consider the following HTML, which is inside a file called document.html.

1
<!doctype HTML>
2
<html class="no-js" lang="">
3
    <head>
4
    <meta charset="utf-8">
5
    <title>A Simple Webpage</title>
6
    <meta name="viewport" content="width=device-width, initial-scale=1">
7
    </head>
8
9
    <body>
10
        <p>Hello <b>world</b>! This is a basic webpage.</p>
11
        <p>Here is a list of some <i>random</i> words:</p>
12
        <ul class="big-words">
13
            <li>Impedimenta</li>
14
            <li>Decompensation</li>
15
            <li>Tergiversation</li>
16
            <li>Transcendentalism</li>
17
            <li>Polyphiloprogenitive</li>
18
        </ul>
19
    </body>
20
</html>
 

We can pass the HTML file as an argument to the PyQuery object, and then we will be able to apply jQuery-style queries to it.

1
webpage = pq(filename = 'document.html')

The value returned by the call to the PyQuery object is similar to what you get with jQuery() or $() when using the jQuery library. Just like the html() method in jQuery, there is an html() method in PyQuery, where you will be able to get or set the HTML content of the selected element.

Currently, the webpage object is representative of the whole document, so it returns the markup of the entire page:

1
print(webpage.html())
2
3
'''

4
<head>

5
<meta charset="utf-8"/>

6
<title>A Simple Webpage</title>

7
<meta name="viewport" content="width=device-width, initial-scale=1"/>

8
</head>

9


10
<body>

11
    <p>Hello <b>world</b>! This is a basic webpage.</p>

12
    <p>Here is a list of some <i>random</i> words:</p>

13
    <ul class="big-words">

14
        <li>Impedimenta</li>

15
        <li>Decompensation</li>

16
        <li>Tergiversation</li>

17
        <li>Transcendentalism</li>

18
        <li>Polyphiloprogenitive</li>

19
    </ul>

20
</body>

21
'''

Let's say you want to get the markup of the first p tag instead of the whole document. What do you do? You simply pass an appropriate selector to our webpage object. Here is an example:

1
print(webpage("p").html())
2
3
'''

4
Hello <b>world</b>! This is a basic webpage.

5
'''

Now take a look at the following code, where we will first set the HTML for our selector using the html() method and then access it using the same html() method.

1
from pyquery import PyQuery as pq
2
3
webpage = pq(filename = 'document.html')
4
5
print(webpage("p").html())
6
'''

7
Hello <b>world</b>! This is a basic webpage.

8
'''
9
10
webpage("p").html("Hello <b>world</b>! I have changed this paragraph.")
11
12
print(webpage("p").html())
13
'''

14
Hello <b>world</b>! I have changed this paragraph.

15
'''

As you can see, it was very easy for us to manipulate the HTML of particular tags. Let's see what else we can change.

Attribute Manipulation Using PyQuery

PyQuery tries to mirror the jQuery API as closely as possible. This means that you get access to an attribute method called attr(), which makes it possible to both get and set the value of attributes for selected DOM nodes.

Our original markup had an ordered list of words. Let's see if we can get the value of the class attribute from the list. We will also use the attr() method to assign an ID to our unordered list. This will require us to pass two different arguments to the method. The first will be the string id, and the second will be the ID that we want to set.

1
from pyquery import PyQuery as pq
2
3
webpage = pq(filename = 'document.html')
4
5
print(webpage("ul").attr('class'))
6
'''

7
big-words

8
'''
9
10
webpage("ul").attr("id", "word-list")
11
12
webpage("p").attr("class", "greeting hello-message")
13
14
print(webpage.html())
15
'''

16
<head>

17
<meta charset="utf-8"/>

18
<title>A Simple Webpage</title>

19
<meta name="viewport" content="width=device-width, initial-scale=1"/>

20
</head>

21


22
<body>

23
    <p class="greeting hello-message">Hello <b>world</b>! This is a basic webpage.</p>

24
    <p class="greeting hello-message">Here is a list of some <i>random</i> words:</p>

25
    <ul class="big-words" id="word-list">

26
        <li>Impedimenta</li>

27
        <li>Decompensation</li>

28
        <li>Tergiversation</li>

29
        <li>Transcendentalism</li>

30
        <li>Polyphiloprogenitive</li>

31
    </ul>

32
</body>

33
'''

As you can see, I have also used the attr() method to add a set of classes to our p elements. Just like its jQuery counterpart, the attr() method in PyQuery also sets the attribute value for all matching elements instead of the first one.

How can we apply the classes only to the first p element? We can simply use the eq() method to get the element at a particular index, as shown below.

1
webpage("p").eq(0).attr("class", "greeting hello-message")

If you are primarily looking to manipulate classes of your elements, you can also consider using the addClass() and removeClass() methods, which will add or remove a CSS class respectively. You can also use the method names add_class() and remove_class() if you are more comfortable working with underscore notation.

Here is an example:

1
webpage("p").eq(0).attr("class", "greeting hello-message")
2
# <p class="greeting hello-message">Hello <b>world</b>! This is a basic webpage.</p>

3
4
webpage("p").eq(0).remove_class("greeting")
5
# <p class="hello-message">Hello <b>world</b>! This is a basic webpage.</p>

6
7
webpage("p").eq(0).add_class("first-message")
8
#  <p class="hello-message first-message">Hello <b>world</b>! This is a basic webpage.</p>

You can also get rid of any attributes assigned to an element by using the remove_attr() method. Keep in mind that there is no add_attr() method as that functionality is served by attr() itself.

Handling CSS Using PyQuery

Apart from attributes, the element would have some CSS properties. To add or modify the CSS properties, you can use the css() method and specify the properties whose values you want to set.

For our example, we will select the bold tag and set its color to dark brown while increasing the font-size to 120%. We will also make the color of our unordered list blue. Here is the code needed to do that:

1
from pyquery import PyQuery as pq
2
3
webpage = pq(filename = 'document.html')
4
5
webpage("ul").css("color", "#1E88E5")
6
webpage("b").css({"font-size": "120%", "color": "#6D4C41"})
7
8
print(webpage)
9
'''

10
<html class="no-js" lang="">

11
    <head>

12
    <meta charset="utf-8"/>

13
    <title>A Simple Webpage</title>

14
    <meta name="viewport" content="width=device-width, initial-scale=1"/>

15
    </head>

16


17
    <body>

18
        <p>Hello <b style="font-size: 120%; color: #6D4C41">world</b>! This is a basic webpage.</p>

19
        <p>Here is a list of some <i>random</i> words:</p>

20
        <ul class="big-words" style="color: #1E88E5">

21
            <li>Impedimenta</li>

22
            <li>Decompensation</li>

23
            <li>Tergiversation</li>

24
            <li>Transcendentalism</li>

25
            <li>Polyphiloprogenitive</li>

26
        </ul>

27
    </body>

28
</html>

29
'''
30
31
with open('updated_markup.html', 'w') as file:
32
    file.write(webpage.outer_html())

As you can see, the css() method in PyQuery is similar to the one in jQuery. After updating the styles, we saved the new markup to a file called updated_markup.html. You can also do the same after making a variety of changes to the markup.

Creating, Removing, & Appending Elements

You might recall that our sample HTML document contains a list of words. Can we expand the list of words? Of course we can. All you need to do is use the append() or prepend() methods. The append() method will append the passed value to the calling node. Similarly, the prepend() method will prepend the passed value to the calling node. Here is an example:

1
from pyquery import PyQuery as pq
2
3
webpage = pq(filename = 'document.html')
4
5
print(webpage("ul"))
6
'''

7
<ul class="big-words">

8
    <li>Impedimenta</li>

9
    <li>Decompensation</li>

10
    <li>Tergiversation</li>

11
    <li>Transcendentalism</li>

12
    <li>Polyphiloprogenitive</li>

13
</ul>

14
'''
15
16
webpage("ul").append("<li>Myrmecophilous</li>")
17
webpage("ul").prepend("<li>Anagnorisis</li>")
18
19
print(webpage("ul"))
20
'''

21
<ul class="big-words">

22
    <li>Anagnorisis</li>

23
    <li>Impedimenta</li>

24
    <li>Decompensation</li>

25
    <li>Tergiversation</li>

26
    <li>Transcendentalism</li>

27
    <li>Polyphiloprogenitive</li>

28
    <li>Myrmecophilous</li>

29
</ul>

30
'''

Another option that you have for appending and prepending elements is the use of the append_to() and prepend_to() methods. The append_to() method will append your calling node to the passed node. Similarly, the prepend_to() method will prepend your calling node to the passed node. However, remember that you cannot simply call these methods on a string. You will have to wrap them in a PyQuery object for the call to work, as shown below:

1
from pyquery import PyQuery as pq
2
3
webpage = pq(filename = 'document.html')
4
5
print(webpage("ul"))
6
'''

7
<ul class="big-words">

8
    <li>Impedimenta</li>

9
    <li>Decompensation</li>

10
    <li>Tergiversation</li>

11
    <li>Transcendentalism</li>

12
    <li>Polyphiloprogenitive</li>

13
</ul>

14
'''
15
16
pq("<li>Myrmecophilous</li>").append_to(webpage("ul"))
17
pq("<li>Anagnorisis</li>").prepend_to(webpage("ul"))
18
19
print(webpage("ul"))
20
'''

21
<ul class="big-words">

22
    <li>Anagnorisis</li>

23
    <li>Impedimenta</li>

24
    <li>Decompensation</li>

25
    <li>Tergiversation</li>

26
    <li>Transcendentalism</li>

27
    <li>Polyphiloprogenitive</li>

28
    <li>Myrmecophilous</li>

29
</ul>

30
'''

As you can see, we get the same output. You can also remove nodes from your document by simply calling the remove() method on them.

Let's do something interesting now. We will alphabetize the list of words by first extracting them, then sorting them, and then reinserting them. Here is the code we use to do that:

1
from pyquery import PyQuery as pq
2
3
webpage = pq(filename = 'document.html')
4
5
print(webpage("ul"))
6
'''

7
<ul class="big-words">

8
    <li>Impedimenta</li>

9
    <li>Decompensation</li>

10
    <li>Tergiversation</li>

11
    <li>Transcendentalism</li>

12
    <li>Polyphiloprogenitive</li>

13
</ul>

14
'''
15
16
words = webpage("ul").children().text().split(' ')
17
words = sorted(words)
18
19
webpage("ul li").remove()
20
21
for word in words:
22
    webpage("ul").append("<li>" + word + "</li>")
23
24
print(webpage("ul"))
25
'''

26
<ul class="big-words">

27
    <li>Decompensation</li>

28
    <li>Impedimenta</li>

29
    <li>Polyphiloprogenitive</li>

30
    <li>Tergiversation</li>

31
    <li>Transcendentalism</li>

32
</ul>

33
'''

Two useful methods that we used in the above example are children() and text(). What do they do? The children() method returns all the elements that are direct children of the calling node. In our case, this means all the list elements. After that, we use the text() method to access the text value of the node.

The extracted text values are then sorted and wrapped inside li tags to append them to our now empty unordered list.

Finding Elements Using PyQuery

There is a good chance that you will be working with HTML documents in order to extract some data from them. Now, before you can extract this data from any element, you will need to locate or find the element.

You can simply use the find() method and pass a selector to it in order to find elements that match that particular selector. This method searches all the descendants of the calling node.

You can also use the closest() method to look for elements if you are interested in searching through the ancestors of that particular selector.

1
from pyquery import PyQuery as pq
2
3
webpage = pq(filename = 'document.html')
4
5
print(webpage.find("b").text())
6
'''

7
world

8
'''
9
10
print(webpage.find("i").closest("p").html())
11
'''

12
Here is a list of some <i>random</i> words:

13
'''

We have already covered the children() method in the previous section. You can also access all the siblings of an element using the siblings() method. Other similar methods that you can use are next_all() and prev_all(), which will give you all the siblings that come next or the siblings that came before respectively. Here is an example:

1
from pyquery import PyQuery as pq
2
3
webpage = pq(filename = 'document.html')
4
5
print(webpage.find("li").eq(2).siblings())
6
'''

7
<li>Decompensation</li>

8
<li>Impedimenta</li>

9
<li>Transcendentalism</li>

10
<li>Polyphiloprogenitive</li>

11
'''
12
13
print(webpage.find("li").eq(2).prev_all())
14
'''

15
<li>Impedimenta</li>

16
<li>Decompensation</li>

17
'''
18
19
print(webpage.find("li").eq(2).next_all())
20
'''

21
<li>Transcendentalism</li>

22
<li>Polyphiloprogenitive</li>

23
'''

Extracting Content From a Webpage

Do you remember when I told you at the beginning of the tutorial that PyQuery can accept input from multiple sources such as a string, a file, or even a URL?

In this section, we will let PyQuery get its markup from a page about Python on Wikipedia. The webpage contains a lot of information about Python. We will try to extract some of it for our consumption. Let's see if we can get all the h2 level headings to keep things simple.

Believe it or not, you only need five lines of code to get your heading text.

1
from pyquery import PyQuery as pq
2
3
webpage = pq(url='https://en.wikipedia.org/wiki/Python_(programming_language)')
4
5
headings = webpage.find("h2 span.mw-headline")
6
7
for heading in headings:
8
    print(pq(heading).text())
9
'''

10
History

11
Design philosophy and features

12
Syntax and semantics

13
Programming examples

14
Libraries

15
Development environments

16
Implementations

17
Development

18
API documentation generators

19
Naming

20
Popularity

21
Uses

22
Languages influenced by Python

23
See also

24
References

25
Further reading

26
External links

27
'''

You might have noticed that I used the selector h2 span.mw-headline instead of using h2. This is because simply using h2 was giving me some additional headings that were not part of the main content. You will also have to do a similar analysis of webpages yourself before determining the appropriate selector to use for extracting the information.

I have already written a tutorial on the Requests module in Python where we used the module to download images. One limitation of the example I included there was that we were hard-coding the path of the image. Let's use the PyQuery library to extract the image paths from a webpage and then feed them to the requests module to download. I will be using the Wikipedia page about the United States for this example:

1
import requests
2
from pyquery import PyQuery as pq
3
4
webpage = pq(url='https://en.wikipedia.org/wiki/United_States')
5
6
images = webpage.find("img.thumbimage")
7
8
for image in images:
9
    image_url = "https:" + pq(image).attr('src')
10
11
    image_name = image_url.split('/')[-1]
12
13
    req = requests.get(image_url)
14
    req.raise_for_status()
15
16
    with open(image_name, 'wb') as fd:
17
        fd.write(req.content)

We don't want to download images of the UI icons, etc. That's why I used a more specific selector to extract our images. I get the image file name by taking the last part of the image path after splitting it along the / character. Here are some of the images that I was able to extract:

United States Image ExtractionUnited States Image ExtractionUnited States Image Extraction

Wrapping It Up

In this tutorial, you saw how to get started with PyQuery, a Python library which allows you to make jQuery queries on XML documents. You saw how to manipulate the attributes and CSS styles of the HTML elements. 

You learnt how to create and append elements to existing elements and insert new elements before and after elements. What you saw in this tutorial is just the tip of the iceberg, and there is a lot more that this library has to offer.

For more detailed information on using this library, I would recommend reading the official documentation.

Advertisement
Did you find this post useful?
Want a weekly email summary?
Subscribe below and we’ll send you a weekly email summary of all new Code tutorials. Never miss out on learning about the next big thing.
Advertisement
Looking for something to help kick start your next project?
Envato Market has a range of items for sale to help get you started.