aiohttp
非同期のhttp(s)の取得のpythonライブラリ
install
$ sudo python3 -m pip install aiohttp
$ sudo python3 -m pip install aiohttp_socks
非同期で取得
マルチプロセスとasyncioを組み合わせると早い
connector = None
headers = {"UserAgent": UserAgent(verify_ssl=False, use_cache_server=False).random}
async with aiohttp.ClientSession(connector=connector, headers=headers) as session:
with timeout(180):
async with session.get(url, ssl=True, params=params) as r:
return await r.text()
socks4,5などを使ったプロキシアクセス
from aiohttp_socks import ProxyType, ProxyConnector, ChainProxyConnector
hostport = random.choice(test_proxy)
connector = ProxyConnector.from_url(f'socks5://AMR338263:cKjELufEUf@{hostport}')
async with aiohttp.ClientSession(connector=connector, headers=headers) as session:
with timeout(180):
async with session.get(url, ssl=True, params=params) as r:
return await r.text()