socket<\/code> which makes it very easy to make network connections and retrive data over those sockets in a Python program.<\/p>\nA socket is much like a file, except that a single socket provides a two-way connection between two programs. You can both read form and write to the same socket. If youu write something to a socket, it is sent to the application at the other end of the socket. If you read from the socket, you are given the data which the other application has sent.<\/p>\n
But if you try to read a socket when the program on the other end of the socket has not sent any data, you just sit and wait. If the programs on both ends of the socket simply wait for some data without sending anything, the will wait for a very long time, so an important part of programs that communicate over the Internet is to have some sort of protocol.<\/p>\n
A protocol is a set of precise rules that determine who is to go first, what they are to do, and then what the responses are to that message, and who sends next, and so on. In a sense the two applications at either end of the socket are doing a dance and making sure not to step on each other's toes.<\/p>\n
There are many documents that describe these network protocols. The Hypertext Transfer Protocol is described in the following document:<\/p>\n
https:\/\/www.w3.org\/Protocols\/rfc2616\/rfc2616.txt<\/a><\/p>\nThis is a long and complex 176-page document with a lot of detail. If you find it interesting, feel free to read it all. But if you take a look around page 36 of RFC2616 you will find the syntax for the GET request. To request a document from a web server, we make a connection, e.g. to the www.pr4e.org<\/code> server on port 80, and then send a line of the form<\/p>\nGET http:\/\/data.pr4e.org\/romeo.txt HTTP\/1.0<\/code><\/pre>\nWhere the second parameter is the web page we are requesting, and then we also send a blank line. The web server will respond with some header information about the document and a blank line followed by the document content.<\/p>\n
The world's simplest web browser<\/h3>\n
Perhaps the easiest way to show how the HTTP protocol works is to write a very simple Python program that makes a connection to a web server and follows the rules of the HTTP protocol to request a document and display what the server sends back.<\/p>\n
import socket\n\nmysock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)\nmysock.connect(('data.pr4e.org', 80))\ncmd = 'GET http:\/\/data.pr4e.org\/romeo.txt HTTP\/1.0\\r\\n\\r\\n'.encode()\nmysock.send(cmd)\n\nwhile True:\n data = mysock.recv(512)\n if len(data) < 1:\n break\n print(data.decode(),end='')\n\nmysock.close()<\/code><\/pre>\nimport socket\n\nmysock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)\nmysock.connect(('blue.yn.cn', 80))\ncmd = 'GET http:\/\/blue.yn.cn\/download\/hello.txt HTTP\/1.0\\r\\n\\r\\n'.encode()\nmysock.send(cmd)\n\nwhile True:\n data = mysock.recv(512)\n if len(data) < 1:\n break\n print(data.decode(),end='')\n\nmysock.close()<\/code><\/pre>\nFirst the program makes a connection to port 80 on the server http:\/\/www.pr4e.com\/<\/a>. Since our program is playing the role of the "web browser", the HTTP protocol says we must send the GET command followed by a blank line. \\r\\n<\/code> signifies an EOL (end of line), so \\r\\n\\r\\n<\/code> signifies nothing between two EOL sequences. That is the equivalent of a blank line.<\/p>\n![\"file\"](\"\/wp-content\/uploads\/2025\/10\/image-1759671193811.png\")
<\/p>\n
A Socket Connection
\nOnce we send that blank line, we write a loop that receives data in 512-character chunks from the socket and prints the data out until there is no more data to read (i.e., the recv() returns an empty string).<\/p>\n
The program produces the following output:<\/p>\n
HTTP\/1.1 200 OK\nDate: Wed, 11 Apr 2018 18:52:55 GMT\nServer: Apache\/2.4.7 (Ubuntu)\nLast-Modified: Sat, 13 May 2017 11:22:22 GMT\nETag: "a7-54f6609245537"\nAccept-Ranges: bytes\nContent-Length: 167\nCache-Control: max-age=0, no-cache, no-store, must-revalidate\nPragma: no-cache\nExpires: Wed, 11 Jan 1984 05:00:00 GMT\nConnection: close\nContent-Type: text\/plain\n\nBut soft what light through yonder window breaks\nIt is the east and Juliet is the sun\nArise fair sun and kill the envious moon\nWho is already sick and pale with grief<\/code><\/pre>\nThe output starts with headers which the web server sends to describe the document. For example, the Content-Type<\/code> header indicates that the document is a plain text document (text\/plain<\/code>).<\/p>\nAfter the server sends us the headers, it adds a blank line to indicate the end of the headers, and then sends the actual data of the file romeo.txt.<\/p>\n
This example shows how to make a low-level network connection with sockets. Sockets can be used to communicate with a web server or with a mail server or many other kinds of servers. All that is needed is to find the document which describes the protocol and write the code to send and receive the data according to the protocol.<\/p>\n
However, since the protocol that we use most commonly is the HTTP web protocol, Python has a special library specifically designed to support the HTTP protocol for the retrieval of documents and data over the web.<\/p>\n
One of the requiments for using the HTTP protocol is the need to send and receive data as bytes objects, instead of strings. In the preceding example, the encode()<\/code> and decode()<\/code> methods convert strings into bytes objects and back again.<\/p>\nThe next example uses b''<\/code> notation to specify that a variable should be stored as a bytes objects. encode()<\/code> and b''<\/code> are equivalent.<\/p>\n>>> b'Hello world'\nb'Hello world'\n>>> 'Hello world'.encode()\nb'Hello world'<\/code><\/pre>\nRetrieving an image over HTTP<\/h3>\n
In the above example, we retrieved a plain text file which had newlines in the file and we simply copied the data to the screen as the program ran. We can use a similar program to retrieve an image aross using HTTP. Instead of copying the data to the screen as the program runs, we accumulate the data in a string, trim off the headers, and then save the image data to a file as follows:<\/p>\n
import socket\nimport time\n\nHOST = 'data.pr4e.org'\nPORT = 80\nmysock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)\nmysock.connect((HOST, PORT))\nmysock.sendall(b'GET http:\/\/data.pr4e.org\/cover3.jpg HTTP\/1.0\\r\\n\\r\\n')\ncount = 0\npicture = b""\n\nwhile True:\n data = mysock.recv(5120)\n if len(data) < 1: break\n #time.sleep(0.25)\n count = count + len(data)\n print(len(data), count)\n picture = picture + data\n\nmysock.close()\n\n# Look for the end of the header (2 CRLF)\npos = picture.find(b"\\r\\n\\r\\n")\nprint('Header length', pos)\nprint(picture[:pos].decode())\n\n# Skip past the header and save the picture data\npicture = picture[pos+4:]\nfhand = open("stuff.jpg", "wb")\nfhand.write(picture)\nfhand.close()<\/code><\/pre>\nWhen the program runs, it produces the following output:<\/p>\n
2824 2824\n2824 5648\n5120 10768\n132 124584\n...\n5120 209332\n5120 214452\n5120 219572\n3524 223096\n2824 225920\n4688 230608\nHeader length 394\nHTTP\/1.1 200 OK\nDate: Mon, 06 Oct 2025 02:09:16 GMT\nServer: Apache\/2.4.52 (Ubuntu)\nLast-Modified: Mon, 15 May 2017 12:27:40 GMT\nETag: "38342-54f8f2e5b6277"\nAccept-Ranges: bytes\nContent-Length: 230210\nVary: Accept-Encoding\nCache-Control: max-age=0, no-cache, no-store, must-revalidate\nPragma: no-cache\nExpires: Wed, 11 Jan 1984 05:00:00 GMT\nConnection: close\nContent-Type: image\/jpeg<\/code><\/pre>\nYou can see that for this url, the Content-Type<\/code> header indicates that body of the document is an image (image\/jpeg<\/code>). Once the program completes, you can view the image data by opening the file stuff.jpg<\/code> in an image viewer.<\/p>\nAs the program runs, you can see that we don't get 5120 characters each time we call the recv()<\/code> method. We get as many characters as have been transferred across the network to us by the web server at the moment we call recv()<\/code>. In this example, we either get as few as 132 characters each time we request up to 5120 characters of data.<\/p>\nYour results may be different depending on your network speed. Also note that on the last call to recv()<\/code> we get 4688 bytes, which is the end of the stream, and in the next call to recv()<\/code> we get a zero-length string that tells us that the server has called close()<\/code> on its end of the socket and there is no more data fothcoming.<\/p>\nWe can slow down our successive recv()<\/code> call s by uncommenting the call to time.sleep()<\/code>. This way, we wait a quarter of a second after each call so that the server can "get ahead" of us and send more data to us before we call recv()<\/code> again. With the delay, in place the program executes as follows:<\/p>\n4236 4236\n5120 9356\n5120 14476\n...\n5120 24716\n5120 229516\n1092 230608\nHeader length 394\nHTTP\/1.1 200 OK\nDate: Mon, 06 Oct 2025 02:28:02 GMT\nServer: Apache\/2.4.52 (Ubuntu)\nLast-Modified: Mon, 15 May 2017 12:27:40 GMT\nETag: "38342-54f8f2e5b6277"\nAccept-Ranges: bytes\nContent-Length: 230210\nVary: Accept-Encoding\nCache-Control: max-age=0, no-cache, no-store, must-revalidate\nPragma: no-cache\nExpires: Wed, 11 Jan 1984 05:00:00 GMT\nConnection: close\nContent-Type: image\/jpeg<\/code><\/pre>\nNow other than the first and last calls to recv()<\/code>, we now get 5120 characters each time we ask for new data.<\/p>\nThere is a buffer between the server making send()<\/code> and our application making recv()<\/code> requests. When we run the program with the delay in place, at some point the server might fill up the buffer in the socket and be forced to pause until our program starts to empty the buffer. The pausing of either the sending application or the receiving application is called "flow control".<\/p>\nRetrieving web pages with urllib<\/code><\/h3>\nWhile we can manually send and receive data over HTTP using the socket library, there is a much simpler way to perform this common task in Python by using the urllib<\/code> library.<\/p>\nUsing urllib<\/code>, you can treat a web page much like a file. You simply indicate which web page you would like to retrieve and urllib<\/code> handles all of the HTTP protocol and header details.<\/p>\nThe equivalent code to read the romeo.txt file from the web using urllib<\/code> is as follows:<\/p>\nimport urllib.request\n\nfhand = urllib.request.urlopen('http:\/\/data.pr4e.org\/romeo.txt')\nfor line in fhand:\n print(line.decode().strip())<\/code><\/pre>\nOnce the web page has been opened with urllib.requests.urlopen<\/code>, we can treat it like a file and read through it using a for<\/code> loop.<\/p>\nWhen the program runs, we only see the output of the contens of the file. The headers are still sent, but the urllib<\/code> code consumes the headers and only returns the data to us.<\/p>\nBut soft what light through yonder window breaks\nIt is the east and Juliet is the sun\nArise fair sun and kill the envious moon\nWho is already sick and pale with grief<\/code><\/pre>\nAs an example, we can write a program to retrieve the data for romeo.txt<\/code> and compute the frequency of each word in the file as follows:<\/p>\nimport urllib.request, urllib.parse, urllib.error\n\nfhand = urllib.request.urlopen('http:\/\/data.pr4e.org\/romeo.txt')\n\ncounts = dict()\nfor line in fhand:\n words = line.decode().split()\n for word in words:\n counts[word] = counts.get(word, 0) + 1\nprint(counts)<\/code><\/pre>\nAgain, once we have opened the web page, we can read it like a local file.<\/p>\n
Reading binary files using urllib<\/code><\/h3>\nSometimes you want to retrieve a non-text (or binary) file such as an image or vidio file. The data in these files is generally not useful to print out, but you can easily make a copy of a URL to a local file on your hard disk using urllib<\/code>.<\/p>\nThe pattern is to open the URL and use read<\/code> to download the entire contents of the document into a string variable (img<\/code>) then write that information to a local file as follows:<\/p>\nimport urllib.request, urllib.parse, urllib.error\n\nimg = urllib.request.urlopen('http:\/\/data.pr4e.org\/cover3.jpg').read()\nfhand = open('cover3.jpg', 'wb')\nfhand.write(img)\nfhand.close()<\/code><\/pre>\nThis program reads all of the data in at once across the network and stores it in the variable img<\/code> in the main memory of your computer, then opens the file cover.jpg<\/code> and writes the data out to your disk. The wb<\/code> argument for open()<\/code> opens a binary file for writing only. This program will work if the size of the file is less than the size of the memory of your computer.<\/p>\nHowever if this is a large audio or video file, this program may crash or at least run extremely slowly when your computer runs out of memory. In order to avoid running out of memory, we retrieve the data in blocks (or buffers) and then write each block to your disk before retrieving the next block. This way the program can read any size file without using up all of the memory you have in your computer.<\/p>\n
import urllib.request, urllib.parse, urllib.error\n\nimg = urllib.request.urlopen('http:\/\/data.pr4e.org\/cover3.jpg')\nfhand = open('cover3.jpg', 'wb')\nsize = 0\nwhile True:\n info = img.read(100000)\n if len(info) < 1: break\n size = size + len(info)\n fhand.write(info)\n\nprint(size, 'characters copied.')\nfhand.close()<\/code><\/pre>\nIn this example, we read only 100,000 characters at a time and then write those characters to the cover3.jpg<\/code> file before retrieving the next 100,000 characters of data from the web.<\/p>\nThis program runs as follows:<\/p>\n
230210 characters copied.<\/code><\/pre>\nParsing HTML and scraping the web<\/h3>\n
One of the common uses of the urllib<\/code> capability in Python is to scrape the web. Web scraping is when we write a program that pretends to be a web browser and retrieves pages, then examines the data in those pages looking for patterns.<\/p>\nAs an example, a search engine such as Google will look at the source of one web page and extract the links to other pages and retrieve those pages, extracting links, and so on. Using this technique, Google spiders its way through nearly all of the pages on the web.<\/p>\n
Google also uses the frequency of links from pages it finds to a particular page as one measure of how "important" a page is and how high the page should appear in its search results.<\/p>\n
Parsing HTML using regular expressions<\/h3>\n
One simple way to parse HTML is to use regular expressions to repeately search for and extract substrings that match a paticular pattern.<\/p>\n
Here is a simple web page:<\/p>\n
<h1>The First Page<\/h1>\n<p>\nIf you like, you can switch to the\n<a href="http:\/\/www.dr-chuck.com\/page2.htm">\nSecond Page<\/a>.\n<\/p><\/code><\/pre>\nWe can construct a well-formed regular expression to match and extract the link values from the above text as follows:<\/p>\n
href="http[s]?:\/\/.+?"<\/code><\/pre>\nOur regular expression looks for strings that start with "href="http:\/\/" or "href="https:\/\/", followed by one or more characters (.+?<\/code>), followed by another dobule quote. The question mark behind the [s]?<\/code> indicates to search for the string "http" followed by zero or one "s".<\/p>\nThe question mark added to the .+?<\/code> indicates that the match is to be done in a "non-greedy" fashion instead of a "greedy" fashion. A non-greedy match tries to find the smallest possible matching string and a greedy match tries to find the largest possible matching string.<\/p>\nWe add parentheses to our regular expression to indicate which part of our matched string we would like to extract, and produce the following program:<\/p>\n
# Search for link values within URL input\nimport urllib.request, urllib.parse, urllib.error\nimport re\nimport ssl\n\n# Ignore SSL certificate errors\nctx = ssl.create_default_context()\nctx.check_hostname = False\nctx.verify_mode = ssl.CERT_NONE\n\nurl = input('Enter - ')\nhtml = urllib.request.urlopen(url, context=ctx).read()\nlinks = re.findall(b'href="(http[s]?:\/\/.*?)"', html)\nfor link in links:\n print(link.decode())<\/code><\/pre>\nThe ssl<\/code> library allows this program to access web sites that strictly enforce HTTPS. The read<\/code> method returns HTML source code as a bytes object instead of returning an HTTPResponse object. The findall<\/code> regular expression method will give us a list of all of the strings that match our regular expression, returning only the link text between the double quotes.<\/p>\nWhen we run the program and input a URL, we get the following output:<\/p>\n
Enter - https:\/\/docs.python.org\nhttps:\/\/docs.python.org\/3\/index.html\nhttps:\/\/www.python.org\/\nhttps:\/\/docs.python.org\/3.8\/\nhttps:\/\/docs.python.org\/3.7\/\nhttps:\/\/docs.python.org\/3.5\/\nhttps:\/\/docs.python.org\/2.7\/\nhttps:\/\/www.python.org\/doc\/versions\/\nhttps:\/\/www.python.org\/dev\/peps\/\nhttps:\/\/wiki.python.org\/moin\/BeginnersGuide\nhttps:\/\/wiki.python.org\/moin\/PythonBooks\nhttps:\/\/www.python.org\/doc\/av\/\nhttps:\/\/www.python.org\/\nhttps:\/\/www.python.org\/psf\/donations\/\nhttp:\/\/sphinx.pocoo.org\/<\/code><\/pre>\nRegular expressions work very nicely when your HTML is well formatted and predictable. But since there are a lot of "broken" HTML pages out there, a solution only using regular expressions might either miss some valid links or end up with bad data.<\/p>\n
This can be solved by using a robust HTML parsing library.<\/p>\n
Parsing HTML using BeautifulSoup<\/h3>\n
Even though HTML looks likes XML and some pages are carefull constructed to be XML, most HTML is generally broken in ways that cause an XML parser to reject the entire page of HTML as improperly formed.<\/p>\n
There are a number of Python libraries which can help you parse HTML and extract data from the pages. Each of the libraries has its strengths and weaknesses and you can pick one based on your needs.<\/p>\n
As an example, we will simply parse some HTML input and extract links using the BeautifulSoup library. BeatifulSoup tolerates highly flawed HTML and still lets you easily extract the data you need. You can download and install the BeautifulSoup code from:<\/p>\n
https:\/\/pypi.python.org\/pypi\/beautifulsoup4<\/a><\/p>\nInformation on installing BeautifulSoup with the Python Package Index tool pip<\/code> is available at:<\/p>\nhttps:\/\/packaging.python.org\/tutorials\/installing-packages\/<\/a><\/p>\npip install beautifulsoup4<\/code><\/p>\nWe will use urllib<\/code> to read the page and then use BeautifulSoup<\/code> to extract the href<\/code> attributes from the anchor (a<\/code>) tags.<\/p>\n# To run this, download the BeautifulSoup zip file\n# http:\/\/www.py4e.com\/code3\/bs4.zip\n# or pip install beautifulsoup4 to ensure you have the latest version\n# and unzip it in the same directory as this file\n\nimport urllib.request, urllib.parse, urllib.error\nfrom bs4 import BeautifulSoup\nimport ssl # defauts to certicate verification and most secure protocol (now TLS)\n\n# Ignore SSL\/TLS certificate errors\nctx = ssl.create_default_context()\nctx.check_hostname = False\nctx.verify_mode = ssl.CERT_NONE\n\nurl = input('Enter - ')\nhtml = urllib.request.urlopen(url, context=ctx).read()\nsoup = BeautifulSoup(html, 'html.parser')\n\n# Retrieve all of the anchor tags\ntags = soup('a')\nfor tag in tags:\n print(tag.get('href', None))<\/code><\/pre>\nThe program prompts for a web address, then opens the web page, reads the data and passes the data to the BeautifulSoup parser, and then retrieves all of the anchor tags and prints out the href<\/code> attribute for each tag.<\/p>\nWhen the program runs, it produces the following output:<\/p>\n
Enter - https:\/\/docs.python.org\ngenindex.html\npy-modindex.html\nhttps:\/\/www.python.org\/\n#\nwhatsnew\/3.6.html\nwhatsnew\/index.html\ntutorial\/index.html\nlibrary\/index.html\nreference\/index.html\nusing\/index.html\nhowto\/index.html\ninstalling\/index.html\ndistributing\/index.html\nextending\/index.html\nc-api\/index.html\nfaq\/index.html\npy-modindex.html\ngenindex.html\nglossary.html\nsearch.html\ncontents.html\nbugs.html\nabout.html\nlicense.html\ncopyright.html\ndownload.html\nhttps:\/\/docs.python.org\/3.8\/\nhttps:\/\/docs.python.org\/3.7\/\nhttps:\/\/docs.python.org\/3.5\/\nhttps:\/\/docs.python.org\/2.7\/\nhttps:\/\/www.python.org\/doc\/versions\/\nhttps:\/\/www.python.org\/dev\/peps\/\nhttps:\/\/wiki.python.org\/moin\/BeginnersGuide\nhttps:\/\/wiki.python.org\/moin\/PythonBooks\nhttps:\/\/www.python.org\/doc\/av\/\ngenindex.html\npy-modindex.html\nhttps:\/\/www.python.org\/\n#\ncopyright.html\nhttps:\/\/www.python.org\/psf\/donations\/\nbugs.html\nhttp:\/\/sphinx.pocoo.org\/<\/code><\/pre>\nThis list is much longer because some HTML anchor tags are relative paths (e.g., tutorial\/index.html) or in-page reference (e.g., '#') that do not include "http:\/\/" or "https:\/\/", which was a requirement in our regular expression.<\/p>\n
You can use also BeautifulSoup to pull out various parts of each tag:<\/p>\n
# To run this, download the BeautifulSoup zip file\n# http:\/\/www.py4e.com\/code3\/bs4.zip\n# and unzip it in the same directory as this file\n\nfrom urllib.request import urlopen\nfrom bs4 import BeautifulSoup\nimport ssl\n\n# Ignore SSL certificate errors\nctx = ssl.create_default_context()\nctx.check_hostname = False\nctx.verify_mode = ssl.CERT_NONE\n\nurl = input('Enter - ')\nhtml = urlopen(url, context=ctx).read()\nsoup = BeautifulSoup(html, "html.parser")\n\n# Retrieve all of the anchor tags\ntags = soup('a')\nfor tag in tags:\n # Look at the parts of a tag\n print('TAG:', tag)\n print('URL:', tag.get('href', None))\n print('Contents:', tag.contents[0])\n print('Attrs:', tag.attrs)<\/code><\/pre>\nEnter - http:\/\/www.dr-chuck.com\/page1.htm\nTAG: <a href="http:\/\/www.dr-chuck.com\/page2.htm">\nSecond Page<\/a>\nURL: http:\/\/www.dr-chuck.com\/page2.htm\nContent: ['\\nSecond Page']\nAttrs: [('href', 'http:\/\/www.dr-chuck.com\/page2.htm')]<\/code><\/pre>\nhtml.parser<\/code> is the HTML parser included in the standard Python 3 library. Information on other HTML parsers is available at:<\/p>\nhttp:\/\/www.crummy.com\/software\/BeautifulSoup\/bs4\/doc\/#installing-a-parser<\/a><\/p>\nThese examples only begin to show the power of BeautifulSoup when it comes to parsing HTML.<\/p>\n
Bonus section for Unix \/ Linux users<\/h3>\n
If you have a Linux, Unix, or Macintosh computer, you probably have commands built in to your operating system that retrieves both plain text and binary files using the HTTP or File Transfer (FTP) protols. One of these commands is curl<\/code>:<\/p>\n$ curl -O http:\/\/www.py4e.com\/cover.jpg<\/code><\/p>\nThe command curl<\/code> is short for "copy URL" and so the two examples listed earlier to retrieve binary files with urllib<\/code> are cleverly named curl1.py<\/code> and curl2.py<\/code> on https:\/\/www.py4e.com\/code3<\/a> as they implement similar functionnality to the curl<\/code> command. There is also a curl3.py<\/code> sample program that does this task a little more effectively, in case you actually want to use this pattern in a program you are writing.<\/p>\n