Skip to content

Commit

Permalink
更新域名模糊匹配
Browse files Browse the repository at this point in the history 
增加了域名模糊匹配功能，考虑到部分用户对这个功能需求不强，于是单独开了一个页面，感知不强。不会影响正常体验。
另外将包名一律修改成为小写，规范化了代码。
同时为多处功能添加注释，删除了一些不必要的代码。优化了部分代码。
  • Loading branch information
@0Chencc
0Chencc committed Jan 21, 2022
1 parent 32f40b1 commit 774d32a
Show file tree
Hide file tree
Showing 11 changed files with 616 additions and 136 deletions.
29 changes: 27 additions & 2 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,10 @@

当使用burp代理时，会从经过burp的流量中抓取域名进行储存。不需要开启，插件启动以及数据库连接之后就会自动拉取。

* 相似域名模糊匹配

会对相似的域名进行匹配，符合正则的就拉取入库

* 支持mysql/sqlite

我们打算思考一下这个工具与后期其他工具的联动，故而默认选择了mysql作为数据库，根据鸭王的反馈，我们又添加了sqlite作为支持。
Expand All @@ -27,6 +31,8 @@
### TODO LIST

- [x] 支持sqlite
- [x] 相似域名模糊匹配
- [ ] url的后缀名筛选
- [ ] 由于仓促赶时间，所以当前的代码可读性是非常差的，会找个时间重构一下代码。

## 使用方法
Expand Down Expand Up @@ -59,10 +65,29 @@

![](img/passiveCollection.png)

### 0x04 工具特点
### 0x04 相似域名收集

使用如下代码进行相似域名匹配，正则在其中。各位有更优秀的正则可以提交issue，届时我们采纳使用。感谢。

```java
for(String s:BurpExtender.currentRootDomainSet){
//思路：考虑将rootdomain进行切割，例如baidu.com使用切割成baidu com，然后对baidu进行相似度匹配
String[] tmp = s.split("\\.");
//通过切割的长度取需要匹配的部分，通过这个来避免当用户设置根域名为www.baidu.com的时候，会匹配成www,baidu的问题，目前直接取baidu,com
String similarRegex = String.format("((?!-)[A-Za-z0-9-]{1,63}(?<!-)\\.)*(?!-)[A-Za-z0-9-]{0,63}%s[A-Za-z0-9-]{0,63}(?<!-)\\.%s",
tmp[tmp.length-2],tmp[tmp.length-1]);
Pattern similarPattern = Pattern.compile(similarRegex);
Matcher matcher = similarPattern.matcher(domain);
return matcher.find();
}
```

![](img/similarDomain.png)

### 0x05 工具特点

根据字段排序功能可以快速筛选出内网IP、相似网段IP以及相似域名，可以根据这些信息自定义域名字典、爆破HOST等，进一步扩大信息收集范围

![](img/features1.png)

![](img/features2.png)
![](img/features2.png)
2 changes: 1 addition & 1 deletion build.gradle
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
version '1.0.3'
version '1.0.4'

buildscript {
ext.kotlin_version = '1.4.10'
Expand Down
Binary file added img/similarDomain.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
26 changes: 17 additions & 9 deletions src/burp/BurpExtender.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,29 +9,37 @@
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.LinkedBlockingQueue;

import Domain.DomainConsumer;
import Domain.DomainProducer;
import UI.BurpDomain;
import UI.ControlSwitch;
import Utils.Config;
import Utils.DBUtil;
import domain.DomainConsumer;
import domain.DomainProducer;
import ui.BurpDomain;
import ui.ControlSwitch;
import utils.Config;
import utils.DBUtil;

public class BurpExtender implements IBurpExtender, ITab, IHttpListener{

private static IBurpExtenderCallbacks callbacks;
// public static HashSet<String> subDomainSet = new HashSet<>();
public static HashMap<String, HashMap<String, String>> subDomainMap = new HashMap<>();
public static HashMap<String, String> urlMap = new HashMap<>();
public static HashMap<String, HashMap<String, String>> similarSubDomainMap = new HashMap<>();
// public static HashMap<String,HashMap<String,String>> similarDomainMap = new HashMap<>();
public static HashMap<String,String> similarUrlMap = new HashMap<>();
public static BlockingQueue<IHttpRequestResponse> inputQueue = new LinkedBlockingQueue<>();
public static BlockingQueue<String> urlQueue = new LinkedBlockingQueue<>();
public static BlockingQueue<String> subDomainQueue = new LinkedBlockingQueue<>();
public static Integer subDomainCount = 0;
public static Integer urlCount = 0;
public static BlockingQueue<String> similarSubDomainQueue = new LinkedBlockingQueue<>();
public static BlockingQueue<String> similarUrlQueue = new LinkedBlockingQueue<>();
public static int subDomainCount = 0;
public static int urlCount = 0;
public static int similarSubDomainCount = 0;
public static int similarUrlCount = 0;
public static DBUtil db;
public static HashSet<String> currentRootDomainSet = new HashSet<>();
// public static HashSet<String> rootSimilarDomainSet = new HashSet<>();
public static HashMap<String,String> config = Config.initDatabaseSetting;
public static DomainConsumer domainConsumer = new DomainConsumer();
public static final String VERSION = "1.0.3";
public static final String VERSION = "1.0.4";
public static final String EXTENSION_NAME = "BurpDomain";
@Override
public void registerExtenderCallbacks(IBurpExtenderCallbacks callbacks){
Expand Down
10 changes: 3 additions & 7 deletions src/Domain/DomainConsumer.java → src/domain/DomainConsumer.java
View file
Original file line number Diff line number Diff line change
@@ -1,13 +1,7 @@
package Domain;
package domain;

import UI.BurpDomain;
import burp.BurpExtender;

import java.security.SecureRandom;
import java.sql.SQLException;
import java.util.HashMap;
import java.util.Map;

//负责将收集的域名信息写入数据库
public class DomainConsumer extends Thread {

Expand All @@ -28,6 +22,8 @@ public static void QueueToResult(){
if(BurpExtender.db.isConnect){
BurpExtender.db.insertSubDomainQueueToDb(BurpExtender.subDomainQueue);
BurpExtender.db.insertUrlQueueToDb(BurpExtender.urlQueue);
BurpExtender.db.insertSimilarSubDomainQueueToDb(BurpExtender.similarSubDomainQueue);
BurpExtender.db.insertSimilarUrlQueueToDb(BurpExtender.similarUrlQueue);
}
}

Expand Down
182 changes: 161 additions & 21 deletions src/Domain/DomainProducer.java → src/domain/DomainProducer.java
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
package Domain;
package domain;

import UI.BurpDomain;
import Utils.Config;
import utils.Config;
import org.apache.commons.text.StringEscapeUtils;
import burp.*;

Expand All @@ -13,6 +12,7 @@
import java.util.HashSet;
import java.util.List;
import java.util.Set;
import java.util.concurrent.BlockingQueue;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

Expand Down Expand Up @@ -77,6 +77,11 @@ public static boolean checkOrigin(List<String> headers){
return false;
}

/**
* 对流量进行搜索分析
* @param message
* @param mode PASSIVE_MODE=0 被动搜索 ACTIVE_MODE=1 主动搜索
*/
public static void handleMessage(IHttpRequestResponse message, int mode){
String reqUrl = helpers.analyzeRequest(message).getUrl().toString();
List<String> headers = helpers.analyzeRequest(message).getHeaders();
Expand All @@ -101,11 +106,24 @@ public static void handleMessage(IHttpRequestResponse message, int mode){
domains.addAll(respDomains);
urls.addAll(respUrls);
}
for (String domain : domains) {handleDomain(domain, mode);}
for (String url : urls) { handleUrl(url);}
//对子域名以及网址进行搜索，如果一个for一次是不是有点笨呢？
for (String domain : domains) {
handleDomain(domain, mode);
//handleSimilarSubDomain(domain,mode);
}
for (String url : urls) {
handleUrl(url);

}
//在这个地方添加similarDomain
}
}

/**
* 提取子域名，如果是主动搜索还会提取ip信息
* @param domain 传入域名进行判断是否为子域名
* @param mode 判断是被动搜索还是主动搜索
*/
public static void handleDomain(String domain, int mode){
if(isSubdomain(domain)){
// subDomainQueue会定时清空，所有子域名会存在subDomainMap，所以还要加个判断
Expand All @@ -122,10 +140,62 @@ public static void handleDomain(String domain, int mode){
data.put("ipAddress", ip);
}
}
}else if(isSimilarSubDomain(domain)){
if(!BurpExtender.similarSubDomainQueue.contains(domain)&&!BurpExtender.similarSubDomainMap.containsKey(domain)&&!BurpExtender.currentRootDomainSet.contains(domain)){
BurpExtender.similarSubDomainQueue.add(domain);
HashMap<String, String> data = new HashMap<>();
BurpExtender.similarSubDomainMap.put(domain, data);
String time = getCurrentTime();
data.put("time", time);
if(mode == ACTIVE_MODE){
String ip = Config.getDomainIp(domain);
data.put("ipAddress", ip);
}
// BurpExtender.similarSubDomainMap.put(domain, data);
}
}
}

public static void handleUrl(String url){
// /**
// * 提取相似域名(暂时没有这个需求，不要使用这个接口)
// * @param domain 传入域名进行判断是否为相似域名
// * @param mode
// * @deprecated
// */
// public static void handleSimilarDomain(String domain,int mode){
// if (isSimilarDomain(domain)){
// BurpExtender.rootSimilarDomainSet.add(domain);
// }
// }

/* *//**
*收集相似域名的子域名
* @param domain
* @param mode
*//*
public static void handleSimilarSubDomain(String domain,int mode){
if(isSimilarSubDomain(domain)){
// subDomainQueue会定时清空，所有子域名会存在subDomainMap，所以还要加个判断
if(!BurpExtender.similarSubDomainQueue.contains(domain)&&!BurpExtender.similarSubDomainMap.containsKey(domain)){
BurpExtender.similarSubDomainQueue.add(domain);
HashMap<String, String> data = new HashMap<>();
BurpExtender.similarSubDomainMap.put(domain, data);
String time = getCurrentTime();
data.put("time", time);
// 如果是主动搜索可以直接获取IP，即使DNS卡住也不会导致burp堵塞
// 通过流量被动收集，会定时获取IP，具体实现在DBUtil.insertSubDomainQueueToDb()中
if(mode == ACTIVE_MODE){
String ip = Config.getDomainIp(domain);
data.put("ipAddress", ip);
}
}
}
}*/
/**
* 提取关联网址
* @param url 传入url进行提取
*/
public static void handleUrl(String url){
try{
URL u = new URL(url);
String domain = u.getHost();
Expand All @@ -136,34 +206,42 @@ public static void handleUrl(String url){
String port = String.valueOf(u.getPort());
url = u.getProtocol() + "://" + u.getHost() + ":" + port + path;
}
url = getFormatURL(url);
if(isSubdomain(domain) && !uselessExtension(path, USELESS_URL_EXTENSIONS)){
if(!BurpExtender.urlQueue.contains(url)&&!BurpExtender.urlMap.containsKey(url)){
BurpExtender.urlQueue.add(url);
BurpExtender.urlMap.put(url, getCurrentTime());
if (isSubdomain(domain)){
if(!uselessExtension(path, USELESS_URL_EXTENSIONS)){
if(!BurpExtender.urlQueue.contains(url)&&!BurpExtender.urlMap.containsKey(url)){
BurpExtender.urlQueue.add(url);
BurpExtender.urlMap.put(url, getCurrentTime());
}
}
}else if (isSimilarSubDomain(domain)&!BurpExtender.currentRootDomainSet.contains(domain)){
if(!uselessExtension(path, USELESS_URL_EXTENSIONS)){
if(!BurpExtender.similarUrlQueue.contains(url)&&!BurpExtender.similarUrlMap.containsKey(url)){
BurpExtender.similarUrlQueue.add(url);
BurpExtender.similarUrlMap.put(url, getCurrentTime());
}
}
}
}catch (Exception e){
BurpExtender.getStderr().println(e);
}
}

public static String getFormatURL(String url){
try{

}catch (Exception e){
BurpExtender.getStderr().println(url);
}
return url;
}
// public static String getFormatURL(String url){
// try{
//
// }catch (Exception e){
// BurpExtender.getStderr().println(url);
// }
// return url;
// }

public static String getCurrentTime(){
SimpleDateFormat formatter= new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
Date date = new Date(System.currentTimeMillis());
return formatter.format(date);
}

public static byte[] subByte(byte[] b,int srcPos,int length){
public static byte[] subByte(byte[] b, int srcPos, int length){
byte[] b1 = new byte[length];
System.arraycopy(b, srcPos, b1, 0, length);
return b1;
Expand Down Expand Up @@ -216,7 +294,11 @@ public static String decodeResp(String resp){
return resp;
}


/**
* 将流量传入到该函数进行解析提取域名
* @param httpResponse
* @return
*/
public static Set<String> grepDomain(String httpResponse) {
Set<String> domains = new HashSet<>();
Pattern pDomainNameOnly = Pattern.compile(DOMAIN_NAME_PATTERN);
Expand Down Expand Up @@ -258,6 +340,11 @@ public static Set<String> grepUrls(String httpResponse){
return urls;
}

/**
* 判断是否为子域名
* @param domain
* @return
*/
public static boolean isSubdomain(String domain){
for (String s : BurpExtender.currentRootDomainSet) {
if(domain.endsWith("."+s)){
Expand All @@ -267,13 +354,66 @@ public static boolean isSubdomain(String domain){
return false;
}

// /**
// * 判断是否为相似域名
// * @param domain
// * @return
// * @deprecated
// */
// public static boolean isSimilarDomain(String domain){
// for(String s:BurpExtender.currentRootDomainSet){
// //思路：考虑将rootdomain进行切割，例如baidu.com使用切割成baidu com，然后对baidu进行相似度匹配
// String[] tmp = s.split("\\.");
// //通过切割的长度取需要匹配的部分，通过这个来避免当用户设置根域名为www.baidu.com的时候，会匹配成www,baidu的问题，目前直接取baidu,com
// String similarRegex = String.format("(?!-)[A-Za-z0-9-]{0,63}%s[A-Za-z0-9-]{0,63}(?<!-).%s",tmp[tmp.length-2],tmp[tmp.length-1]);
// Pattern similarPattern = Pattern.compile(similarRegex);
// Matcher matcher = similarPattern.matcher(domain);
// return matcher.find();
// }
// return false;
// }

/**
* 判断是否为相似域名的子域名
* @param domain
* @return
*/
public static boolean isSimilarSubDomain(String domain){
for(String s:BurpExtender.currentRootDomainSet){
//思路：考虑将rootdomain进行切割，例如baidu.com使用切割成baidu com，然后对baidu进行相似度匹配
String[] tmp = s.split("\\.");
//通过切割的长度取需要匹配的部分，通过这个来避免当用户设置根域名为www.baidu.com的时候，会匹配成www,baidu的问题，目前直接取baidu,com
String similarRegex = String.format("((?!-)[A-Za-z0-9-]{1,63}(?<!-)\\.)*(?!-)[A-Za-z0-9-]{0,63}%s[A-Za-z0-9-]{0,63}(?<!-)\\.%s",
tmp[tmp.length-2],tmp[tmp.length-1]);
Pattern similarPattern = Pattern.compile(similarRegex);
Matcher matcher = similarPattern.matcher(domain);
return matcher.find();
}
// for(String s:BurpExtender.rootSimilarDomainSet){
// if (similarDomain.endsWith("."+s)){
// return true;
// }
// }
return false;
}

/**
* 判断是否为url编码
* @param line
* @return
*/
public static boolean urlCode(String line) {
String patternRule = "(%(\\p{XDigit}{2}))";
Pattern pattern = Pattern.compile(patternRule);
Matcher matcher = pattern.matcher(line.toLowerCase());
return matcher.find();
}

/**
* 判断是否为unicode
* @param line
* @return
*/
public static boolean unicodeCode(String line) {
String patternRule = "(\\\\u(\\p{XDigit}{4}))";
Pattern pattern = Pattern.compile(patternRule);
Expand Down
Loading

0 comments on commit 774d32a

Please sign in to comment.