AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |
Back to Blog
Requests python download file9/9/2023 ![]() However, for files over 100M, almost all the files stored locally are inconsistent with the content-length in the obtained response header. Total_size = int(response.headers)įor chunk in er_content(chunk_size=102400):Īfter downloading, I will use the following method to compare with the above total_size with open('xxx', 'r') as f: Response = requests.get(url, stream=True) ![]() This generates unnecessary traffic and load on that server which is a pretty crappy thing to do The nicer and Pythonic thing to do is to have a separate script that does the request once and saves the required data to a local file. Relevant codes are summarized as follows: _RETRIES = 5 Every time you run that script with your requests.get code in place, you’re making a call to the target web server. ![]() The probability is very high and should not be caused by network reasons. At present, I have a script to obtain files from the CDN chain as backup.Īt present, there is a situation that when using requests to download large files, the downloaded files will be incomplete.
0 Comments
Read More
Leave a Reply. |