WebCollector 教程——接入阿布云代理

最近用了阿布云代理,发现真心良心商家,性价比极高。用下面这个WebCollector的Requester插件(Http请求插件)就可直接在WebCollector中使用阿布云代理。

阿布云代理官方:https://www.abuyun.com/


package cn.edu.hfut.dmic.webcollector.plugin.requester; import cn.edu.hfut.dmic.webcollector.model.CrawlDatum; import cn.edu.hfut.dmic.webcollector.net.HttpRequest; import cn.edu.hfut.dmic.webcollector.net.HttpResponse; import cn.edu.hfut.dmic.webcollector.net.Requester; import java.net.Authenticator; import java.net.InetSocketAddress; import java.net.PasswordAuthentication; import java.net.Proxy; /** * WebCollector使用阿布云代理的Http请求插件 * * 插件使用方法: * crawler.setRequester(new AbuyunDynamicProxyRequester("阿布云动态代理用户名", "阿布云动态代理密码")); * 一行代码解决,无需自定义代理池及其它代理切换规则 * * 阿布云代理官网: * https://www.abuyun.com/ * * @author hu */ public class AbuyunDynamicProxyRequester implements Requester { /** * * @param proxyUser 阿布云动态代理用户名 * @param proxyPass 阿布云动态代理密码 */ public AbuyunDynamicProxyRequester(String proxyUser, String proxyPass) { Authenticator.setDefault(new ProxyAuthenticator(proxyUser, proxyPass)); } @Override public HttpResponse getResponse(CrawlDatum crawlDatum) throws Exception { HttpRequest request = new HttpRequest(crawlDatum.url()); String proxyServer = "proxy.abuyun.com"; int proxyPort = 9020; InetSocketAddress addr = new InetSocketAddress(proxyServer, proxyPort); Proxy proxy = new Proxy(Proxy.Type.HTTP, addr); request.setProxy(proxy); return request.response(); } public static class ProxyAuthenticator extends Authenticator { private String user, password; public ProxyAuthenticator(String user, String password) { this.user = user; this.password = password; } protected PasswordAuthentication getPasswordAuthentication() { return new PasswordAuthentication(user, password.toCharArray()); } } }

阿布云动态代理插件使用方法:


crawler.setRequester(new AbuyunDynamicProxyRequester("阿布云动态代理用户名", "阿布云动态代理密码"));

You may also like...

发表评论

电子邮件地址不会被公开。 必填项已用*标注

推荐文章
加QQ群下载完整Eclipse项目

WebCollector网络爬虫:250108697(已满)

WebCollector网络爬虫2群:345054141

热门