小弟新手，希望各位大神帮个关于httpclient登陆微博的忙
小弟java菜鸟，希望能抓取新浪微博中的页面，可是登陆问题不知怎么解决，百度之后得知httpclient可以解决登陆问题，故希望各位大神能给一段代码教教我怎么实现，最好需要哪些包也一并告诉我，万分感谢~~ Java 新浪微博 --------------------编程问答-------------------- 单纯的抓取界面内容的话。用URL获取html页面就可以了。。 --------------------编程问答-------------------- 直接抓取的东西和从浏览器上取网页源代码的东西不一样，好像是因为没有登陆信息，返回的responseCode是500
--------------------编程问答--------------------
直接抓取的东西和从浏览器上取网页源代码的东西不一样，好像是因为没有登陆信息，返回的responseCode是500,求大神指导 --------------------编程问答-------------------- httpclient是绝对可以的，首先你要知道新浪微博的登录过程，“审查元素”到网络那part，查看login首先会有一个prelogin，这个是用来计算当前时间的，因为新浪微博的账户、密码是需要加密的，这个prelogin就是那一系列的参数呗，然后密码的加密很繁琐，好像是先RSA，再根据时间
加密，再RSA，你直接找到他密码加密的js用java的scriptengine解析后模仿新浪微博登录进去就行了，之后你就获得了一个cookie，通过这个cookie你就可以以登录的状态爬取各种微博信息啦 --------------------编程问答-------------------- 大神你说的太难懂了，我是菜鸟，能请您给出一个实现的代码吗？最好有点注释，我不会直接抄的，关键您说的我真是不知道都代表了什么。 --------------------编程问答--------------------
无意间才看到你这个哈，那就把登录部分的代码贴一下吧，还有，俺不是大神，也就今年才毕业而已，代码是新浪微博更新了V2.0之后改过的，好久没弄过网页版的新浪了，最近又奔手机玩去了，呵呵，如果有问题的话你就跟新一下js文件，保持那个js是新浪的最新js就OK了，很简单的我就不说了哈
package cn.com.leku.service;
import java.io.IOException;
import java.io.UnsupportedEncodingException;
import java.net.URLEncoder;
import java.util.ArrayList;
import java.util.Date;
import java.util.List;
import org.apache.commons.codec.binary.Base64;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.http.HttpResponse;
import org.apache.http.NameValuePair;
import org.apache.http.ParseException;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.HttpClient;
import org.apache.http.client.entity.UrlEncodedFormEntity;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.client.params.CookiePolicy;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.message.BasicNameValuePair;
import org.apache.http.params.CoreConnectionPNames;
import org.apache.http.util.EntityUtils;
import cn.com.leku.domain.PreLoginInfo;
import cn.com.leku.sina.SinaUtil;
import cn.com.leku.utils.BigIntegerRSA;
import cn.com.leku.utils.JsonUtil;
import cn.com.leku.utils.PropertyReader;
public class SinaWeiboServiceImpl implements WeiboService {
private static final Log _LOG = LogFactory.getLog(SinaWeiboServiceImpl.class);
/**
* 新浪微博属性路径
*/
private final static String SINA_PROPERTIES_PATH = SinaUtil.class.getClassLoader().getResource("sina.properties").getPath();
/**
* 内容匹配的JS开头
*/
protected final static String CONTENT_PATTERN = "<script>STK && STK.pageletM && STK.pageletM.view({\"pid\":\"pl_content_hisFeed\"";
/**
* 人物信息匹配的JS开头
*/
protected final static String INFO_PATTERN = "<script>STK && STK.pageletM && STK.pageletM.view({\"pid\":\"pl_profile_hisInfo\"";
/**
* 头像、粉丝数等匹配的开头
*/
protected final static String PHOTO_PATTERN = "<script>STK && STK.pageletM && STK.pageletM.view({\"pid\":\"pl_profile_photo\"";
/**
* 导航TAB匹配的JS开头
*/
protected final static String NAV_PATTERN = "<script>STK && STK.pageletM && STK.pageletM.view({\"pid\":\"pl_profile_nav\"";
/**
* 关注人detaiL
*/
protected final static String HIS_FOLLOW_PATTERN = "<script>STK && STK.pageletM && STK.pageletM.view({\"pid\":\"pl_relation_hisFollow\"";
/**
* 粉丝detail
*/
protected final static String HIS_FANS_PATTERN = "<script>STK && STK.pageletM && STK.pageletM.view({\"pid\":\"pl_relation_hisFans\"";
/**
* 个人信息URL匹配的连接开头
*/
protected final static String TAGS_URL_PREFIX = "/find/f?";
/**
* 默认编码
*/
protected final static String DEFAULT_ENCODING = PropertyReader.getValue(SINA_PROPERTIES_PATH, "sina.global.encoding");
public HttpClient doLogin() {
// TODO Auto-generated method stub
HttpClient client =null;
HttpPost post = null;
HttpResponse response = null;
HttpGet getMethod = null;
try {
client = new DefaultHttpClient();
// 设置client的默认参数
client.getParams().setParameter("http.protocol.cookie-policy", CookiePolicy.BROWSER_COMPATIBILITY);
client.getParams().setParameter(CoreConnectionPNames.CONNECTION_TIMEOUT, Integer.parseInt(PropertyReader.getValue(SINA_PROPERTIES_PATH, "sina.connection.timeout")));
// 进行登录请求
post = new HttpPost(PropertyReader.getValue(SINA_PROPERTIES_PATH, "sina.address.php.login"));
post.setEntity(new UrlEncodedFormEntity(generateParams(generatePreLoginInfo(client)), DEFAULT_ENCODING));
response = client.execute(post);
String entity = EntityUtils.toString(response.getEntity());
String url = entity.substring(entity.indexOf("http://weibo.com/ajaxlogin.php?"), entity.indexOf("code=0") + 6);
// 获取到实际url进行连接
getMethod = new HttpGet(url);
response = client.execute(getMethod);
entity = EntityUtils.toString(response.getEntity());
//entity = entity.substring(entity.indexOf("userdomain") + 13, entity.lastIndexOf("\""));
} catch (ClientProtocolException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (ParseException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} finally {
if(null != getMethod) {
getMethod.releaseConnection();
}
}
return client;
}
/**
* SSO登录的附加参数
* @param info
* @return
*/
private List<NameValuePair> generateParams(PreLoginInfo info) {
//SSO登录的附加参数
List<NameValuePair> nvps = new ArrayList<NameValuePair>();
nvps.add(new BasicNameValuePair("entry", PropertyReader.getValue(SINA_PROPERTIES_PATH, "sina.nameValuePair.entry")));
nvps.add(new BasicNameValuePair("gateway", PropertyReader.getValue(SINA_PROPERTIES_PATH, "sina.nameValuePair.gateway")));
nvps.add(new BasicNameValuePair("from", PropertyReader.getValue(SINA_PROPERTIES_PATH, "sina.nameValuePair.from")));
nvps.add(new BasicNameValuePair("savestate", PropertyReader.getValue(SINA_PROPERTIES_PATH, "sina.nameValuePair.savestate")));
nvps.add(new BasicNameValuePair("useticket", PropertyReader.getValue(SINA_PROPERTIES_PATH, "sina.nameValuePair.useticket")));
nvps.add(new BasicNameValuePair("ssosimplelogin", PropertyReader.getValue(SINA_PROPERTIES_PATH, "sina.nameValuePair.ssosimplelogin")));
nvps.add(new BasicNameValuePair("vsnf", PropertyReader.getValue(SINA_PROPERTIES_PATH, "sina.nameValuePair.vsnf")));
nvps.add(new BasicNameValuePair("su", encodeAccount(PropertyReader.getValue(SINA_PROPERTIES_PATH, "sina.accout"))));
nvps.add(new BasicNameValuePair("service", PropertyReader.getValue(SINA_PROPERTIES_PATH, "sina.nameValuePair.service")));
nvps.add(new BasicNameValuePair("servertime", String.valueOf(info.getServertime())));
nvps.add(new BasicNameValuePair("nonce", info.getNonce()));
nvps.add(new BasicNameValuePair("pwencode", PropertyReader.getValue(SINA_PROPERTIES_PATH, "sina.nameValuePair.pwencode")));
nvps.add(new BasicNameValuePair("rsakv", info.getRsakv()));
nvps.add(new BasicNameValuePair("prelt", PropertyReader.getValue(SINA_PROPERTIES_PATH, "sina.nameValuePair.prelt")));
nvps.add(new BasicNameValuePair("sp", new BigIntegerRSA().rsaCrypt(PropertyReader.getValue(SINA_PROPERTIES_PATH, "sina.password"), info.getNonce(), info.getServertime(), info.getPubkey())));
nvps.add(new BasicNameValuePair("url", PropertyReader.getValue(SINA_PROPERTIES_PATH, "sina.nameValuePair.url")));
nvps.add(new BasicNameValuePair("returntype", PropertyReader.getValue(SINA_PROPERTIES_PATH, "sina.nameValuePair.returntype")));
nvps.add(new BasicNameValuePair("encoding", PropertyReader.getValue(SINA_PROPERTIES_PATH, "sina.nameValuePair.encoding")));
return nvps;
}
/**
* 对账户进行编码
* @param account 账户
* @return
*/
private String encodeAccount(String account) {
String userName = "";
try {
userName = Base64.encodeBase64String(URLEncoder.encode(account, DEFAULT_ENCODING).getBytes());
} catch (UnsupportedEncodingException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return userName;
}
private static String getServerTime() {
// TODO Auto-generated method stub
long servertime = new Date().getTime() / 1000;
return String.valueOf(servertime);
}
/**
* 进行新浪微博的预登录<BR>
* 获取相应的时间、登录参数等信息用以计算密码
* @param client 客户端
* @return
* @throws ClientProtocolException
* @throws IOException
*/
private PreLoginInfo generatePreLoginInfo(HttpClient client) throws ClientProtocolException, IOException {
// 预登录处理，用于获取准确的登录时间，和计算的参数
String preloginurl = PropertyReader.getValue(SINA_PROPERTIES_PATH, "sina.address.php.prelogin") + getServerTime();
HttpGet httpGet = new HttpGet(preloginurl);
HttpResponse response = client.execute(httpGet);
String getResp = EntityUtils.toString(response.getEntity());
int firstLeftBracket = getResp.indexOf("(");
int lastRightBracket = getResp.lastIndexOf(")");
String jsonBody = getResp.substring(firstLeftBracket + 1, lastRightBracket);
return JsonUtil.getInstance().json2Obj(jsonBody, PreLoginInfo.class);
}
@Override
public void crawl(HttpClient client, String url, int typeId) {
// TODO Auto-generated method stub
}
@Override
public void parser() {
// TODO Auto-generated method stub
}
}
这个是未优化版的，我稍微改了下原来的项目，虽然原来的项目也是单挑的，但是没有公司的版权许可我就不完整的弄出来了，其实也就删了线程池爬取优化和文件存储优化两部分而已，然后是比较重要的加密类 --------------------编程问答--------------------
package cn.com.leku.utils;
import javax.script.Invocable;
import javax.script.ScriptEngine;
import javax.script.ScriptEngineManager;
import javax.script.ScriptException;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
public class BigIntegerRSA {
private static final Log _LOG =LogFactory.getLog(BigIntegerRSA.class);
/**
* 调用js引擎获取加密后的密码字符串
* @param password 用户密码
* @param nonce
* @param serviceTime 服务器时间
* @param pk
* @return js引擎获取加密后的密码字符串
*/
public String rsaCrypt(String password, String nonce, long serviceTime, String pk) {
//调用js引擎解析
ScriptEngineManager sem = new ScriptEngineManager();
ScriptEngine se = sem.getEngineByName("javascript");
try {
se.eval(getJs());
} catch (ScriptException e) {
e.printStackTrace();
_LOG.warn("解析js出错", e);
}
String pass = "";
if (se instanceof Invocable) {
Invocable invoke = (Invocable) se;
try {
pass = invoke.invokeFunction("getpass", password, serviceTime, nonce, pk).toString();
} catch(NoSuchMethodException e1) {
_LOG.warn("js中不存在对应的方法", e1);
} catch (ScriptException e2) {
// TODO Auto-generated catch block
_LOG.warn("解析js出错", e2);
}
}
return pass;
}
// rsa2加密
public static String getJs() {
String js = ""+
// 自定义function 获取密码
"function getpass(pwd,servicetime,nonce,rsaPubkey){var RSAKey=new sinaSSOEncoder.RSAKey();RSAKey.setPublic(rsaPubkey,'10001');var password=RSAKey.encrypt([servicetime,nonce].join('\\t')+'\\n'+pwd);return password;}";
return js;
}
}
最后一个方法那里懒得改了，直接找了sina的js复制过来格式化了一下，不过LZ最好再用httpclient去动态的获取这个js文件，然后读取js文件就好了，这个算是公司技术隐私的一部分我也不完整的发上来了哈
补充：Java , Web 开发
