Official Website: https://www.nstl.gov.cn/
Words Website: https://www.nstl.gov.cn/stkos.html?t=Concept&q=
- Usage Demo
- Other Useful Resources
- Prevent Anti-reptile
- Links of the Contents
- ID Order
- Parallelism Crawler
- Announcement
- License
-
First of all, get all the words' IDs from the main page
You can request this website instead to get the IDs from the page. You can change "&pageNumber=1" to "&pageNumber=100" to get the contents from Page 100. Please follow the codes "get-website-IDs.ipynb".
-
After get the IDs of words, use "fast-crawler-NSTL-data.ipynb" to download and capture all the contents from the websites.
The words will be saved as JSON format.
Good luck to your future reptile!
-
Python selenium package (5 - 20 seconds for a website)
1). Don't recommend this method if there are tons of websites
2). You need to download Chrome Driver to capture the websites.
3). If you have (a) few dynamic websites, you can use "selenium-reptile-script.py" script. This file can be used as a Reference. After a little corrections, you can use this file.
-
Google Developer Tools -> Network -> Headers -> Request URL (less than one second for a website)
1). Suggest to use this method.
2). Use "fast-reptile-script-YEAR2018.py" script or follow "fast-crawler-NSTL-data.ipynb" script.
-
Use a fake device instead of visiting directly
# A fake device to avoid the Anti reptile USER_AGENTS = [ "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; AcooBrowser; .NET CLR 1.1.4322; .NET CLR 2.0.50727)", "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; Acoo Browser; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506)", "Mozilla/4.0 (compatible; MSIE 7.0; AOL 9.5; AOLBuild 4337.35; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)", "Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US)", "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET CLR 2.0.50727; Media Center PC 6.0)", "Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET CLR 1.0.3705; .NET CLR 1.1.4322)", "Mozilla/4.0 (compatible; MSIE 7.0b; Windows NT 5.2; .NET CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.2; .NET CLR 3.0.04506.30)", "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN) AppleWebKit/523.15 (KHTML, like Gecko, Safari/419.3) Arora/0.3 (Change: 287 c9dfb30)", "Mozilla/5.0 (X11; U; Linux; en-US) AppleWebKit/527+ (KHTML, like Gecko, Safari/419.3) Arora/0.6", "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.2pre) Gecko/20070215 K-Ninja/2.1.1", "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.9) Gecko/20080705 Firefox/3.0 Kapiko/3.0", "Mozilla/5.0 (X11; Linux i686; U;) Gecko/20070322 Kazehakase/0.4.5", "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8) Gecko Fedora/1.9.0.8-1.fc10 Kazehakase/0.5.6", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/535.20 (KHTML, like Gecko) Chrome/19.0.1036.7 Safari/535.20", "Opera/9.80 (Macintosh; Intel Mac OS X 10.6.8; U; fr) Presto/2.9.168 Version/11.52", ] random_agent = USER_AGENTS[randint(0, len(USER_AGENTS) - 1)] headers = { 'User-Agent': random_agent, }
-
Add a try-except-sleep to avoid "Refuse connection" and use timeout=(5, 10) to avoid Python "No response" Bug
for j in range(10): try: res = requests.get(url, headers=headers, verify=False, timeout=(5, 10)) contents = res.text except Exception as e: if j >= 9: print('The exception has happened', '-' * 100) else: time.sleep(0.5) else: time.sleep(0.5) break
-
Use SSL package and "verify=False" to disable Network Certificate
import ssl # Avoid SSL Certificate to access the HTTP website ssl._create_default_https_context = ssl._create_unverified_context
or
res = requests.get(url, headers=headers, verify=False, timeout=(5, 10)) contents = res.text
-
Use urllib3 package to disable warnings
import urllib3 # Disable all kinds of warnings urllib3.disable_warnings()
-
Block current IP address
1). Use a VPN software to change the IP address.
2). Reptile the websites at home.
3). Wait for 2 days to reptile if it's not urgent.
I use an Example here, ID is C018781660:
To get English Term + Chinese Term + Synonyms: Link
To get Fields: Link
To get IDs in a single page (~ 10 IDs in one page): Link
In fact, there is an order for the NSTL words w.r.t. different years, e.g., 2018, 2019, 2020, but I don't recommend you to use the order method because you might miss some words in this way.
In contrast, I think it is superior to capture all the words' IDs first via this script, and then capture the contents w.r.t. these word IDs via this script.
All the words' IDs were crawlled via this script and available as a CSV file.
Orders for NSTL words:
YEAR 2020: C0200 + {29329 - 55000} --> 19,960 Words
YEAR 2019: C019 + {000000 - 500000} --> 395,093 Words
YEAR 2018: C018 + {781660 - 999999} --> 199,906 Words
FYI, there are lots of blank pages in these IDs' website. You can look through and run the codes, and then you will find out.
Until August 18th, 2020, there are 614,959 Words available.
The crawled dataset (JSON Format) is available at the Shared Google Drive.
Main idea:
-> import threading
import threading-> define a function (e.g., fun)
def fun(param)
***
***
return ***-> Make the function as one thread
task = threading.Thread(target=fun, args=param)-> Start the task
task.start()Notice: The number of parallel crawlers should be twice as your number of CPU cores
For example, if you are using a four-core CPU, I'd suggest you to start 8 tasks.
A Python demo is available below (script code demo):
import threading
def get_JSON(ID: str):
save_year = 'YEAR-20/'
for i in ID:
JSON_file = ''
# Get the contents of the website
contents = read_url(ID=i)
if "Concept" in contents:
# Find the English Term from the contents
Eng_term, con_cut_eng = find_English_term(content=contents)
# Find the Chinese Term from the contents
Chi_term, con_cut_chi = find_Chinese_term(content=con_cut_eng)
# Find the English Definition from the contents
Eng_def, con_cut_def = find_English_definition(content=con_cut_chi)
# Find the Synonym Words from the contents
synonym_word = synonym(content=con_cut_chi)
# Find the Fields from another contents
field_names = field(ID=i)
# Combine all the found data and make string for JSON
JSON_file += '{'
JSON_file += '"English Term": ["'
JSON_file += Eng_term
JSON_file += '"], '
JSON_file += '"Chinese Term": ["'
JSON_file += Chi_term
JSON_file += '"], '
JSON_file += '"English Definition": ["'
JSON_file += Eng_def
JSON_file += '"], '
JSON_file += '"Synonym Words": ['
JSON_file += synonym_word
JSON_file += '], '
if field_names == ' ':
JSON_file += field_names
else:
JSON_file += '"Fields": []}'
# Save the JSON File for each word
save_json(eval(JSON_file), save_year + '%s_word.json' % i)
print('The %s word of %s has been successfully saved!' % (i, save_year))
else:
print(i)
print('There was no data in this website!')
# The main function
if __name__ == '__main__':
IDs = read_csv(csv_path='YEAR_2020.csv')
num = int(np.shape(IDs)[0] / 8)
ID1 = IDs[0 * num: 1 * num]
ID2 = IDs[1 * num: 2 * num]
ID3 = IDs[2 * num: 3 * num]
ID4 = IDs[3 * num: 4 * num]
ID5 = IDs[4 * num: 5 * num]
ID6 = IDs[5 * num: 6 * num]
ID7 = IDs[6 * num: 7 * num]
ID8 = IDs[7 * num:]
threadl = []
task1 = threading.Thread(target=get_JSON, args=(ID1, ))
task2 = threading.Thread(target=get_JSON, args=(ID2, ))
task3 = threading.Thread(target=get_JSON, args=(ID3, ))
task4 = threading.Thread(target=get_JSON, args=(ID4, ))
task5 = threading.Thread(target=get_JSON, args=(ID5, ))
task6 = threading.Thread(target=get_JSON, args=(ID6, ))
task7 = threading.Thread(target=get_JSON, args=(ID7, ))
task8 = threading.Thread(target=get_JSON, args=(ID8, ))
task1.start()
task2.start()
task3.start()
task4.start()
task5.start()
task6.start()
task7.start()
task8.start()🏆 That's all! Enjoy your Python Crawlering and have a wonderful journey!
These scripts and methods were mainly used to capture words contents from the NSTL word websites. However, I believe they can be transfered for other dynamic web crawlering besides the NSTL words. So, enjoy your python crawlering and have a wonderful journey!
MIT License