Resolved: problem finding links in all li tag under ul tag

Question:

I’m trying to get the links in all li tags under the ul tag
HTML code:

<div id="chapter-list" class="sbox" style="">
<ul>
<li>
<a href="https://example.com/manga/name/2">
<div class="chpbox">
<span class="chapternum">
Chapter 2 </span>
</div>
</a>
</li>
<li>
<a href="https://example.com/manga/name/1">
<div class="chpbox">
<span class="chapternum">
Chapter 1 </span>
</div>
</a>
</li>
</ul>
</div>

The code I wrote:

from bs4 import BeautifulSoup
import requests

html_page = requests.get('https://example.com/manga/name/')

soup = BeautifulSoup(html_page.content, 'html.parser')
chapters = soup.find('div', {"id": "chapter-list"})

children = chapters.findChildren("ul" , recursive=False) # when printed, it gives the the whole ul content

for litag in children.find('li'):
    print(litag.find("a")["href"])

When I try to print the li tags links, it gives the following error:

Traceback (most recent call last):
  File "C:\0.py", line 12, in <module>
    for litag in children.find('li'):
  File "C:\Users\hs\AppData\Local\Programs\Python\Python310\lib\site-packages\bs4\element.py", line 2289, in __getattr__
    raise AttributeError(
AttributeError: ResultSet object has no attribute 'find'. You're probably treating a list of elements like a single element. Did you call find_all() when you meant to call find()?

Answer:

You can use find to find the ul in the chapter list. And then find_all to find the list items in the ul. Finally, use find_all again to find the links in each list item and print the URL. Details of these two methods can be found in find and find_all method documentation on bs4. You can use the get_text() after searching by the class chapternum on each link to get the link’s text like Chapter 1. Searching by class be found in bs4 documentation for searching element by class
(Updated) Code:

from bs4 import BeautifulSoup

html_doc = """
<div id="chapter-list" class="sbox" style="">
    <ul>
        <li>
            <a href="https://example.com/manga/name/2">
                <div class="chpbox">
<span class="chapternum">
Chapter 2 </span>
                </div>
            </a>
        </li>
        <li>
            <a href="https://example.com/manga/name/1">
                <div class="chpbox">
<span class="chapternum">
Chapter 1 </span>
                </div>
            </a>
        </li>
    </ul>
</div>
"""

soup = BeautifulSoup(html_doc, 'html.parser')
chapters = soup.find('div', {"id": "chapter-list"})

list_items = chapters.find('ul').find_all('li')

for list_item in list_items:
    for link in list_item.find_all('a'):
        title = link.find('span', class_='chapternum').get_text().strip()
        href = link.get("href")
        print(f"{title}: {href}")

Output:

Chapter 2: https://example.com/manga/name/2
Chapter 1: https://example.com/manga/name/1

References:

If you have better answer, please add a comment about this, thank you!

Resolved: problem finding links in all li tag under ul tag

Resolved: CMake path for python3 libraries doesn’t change (windows 10)

Resolved: GLMM with quasi-Poisson distribution

Resolved: Is it possible to do custom preinstall scripts for npm from which one is run depending on the environment?

Leave A Reply