HaarpCache

Maintainer

• [Kei Kurono] (https://perufw.wordpress.com) <[email protected]>

Table of contents

• Introduction
• Installing
• Reinstall or Update
• Monitoring Logs
• List of Plugins
• Changelog

Introduction

Proxy HaarpCache is a fork of ThunderCache.

HaarpCache is a robust static and dynamic cache that provides support to the DASH technology.

Installing

• You can to install HaarpCache, in distributions Debian/Ubuntu, using a bash script:

    wget https://raw.githubusercontent.com/keikurono/haarpcache/master/installHaarp.sh
    chmod +x installHaarp.sh
    bash installHaarp.sh
  

  Video installation:

    http://www.youtube.com/watch?v=ZdSrxnjY2OA
  

  More information (Installation with Mikrotik, etc):

    https://perufw.wordpress.com/2015/10/28/new-version-haarpcache-v2-0b/
  

• For manual installation:

    su
    apt-get update 
    apt-get upgrade
    apt-get install build-essential mysql-server mysql-client php5 apache2 php5-mysql \
    libblkid-dev libcurl4-gnutls-dev libmysqlclient15-dev libapache2-mod-auth-mysql \
    libapache2-mod-php5 sharutils curl autoconf squid3 git g++-4.4
    cd /usr/src/
    git clone https://github.com/keikurono/haarpcache.git
    cd haarpcache
    ./configure
    ./configure CXX=g++-4.4 # <--- for ubuntu
    make
    make install
    mysql -u root -p < haarp.sql
    cd /etc/init.d
    update-rc.d haarp defaults 98
  

• Create a user for database of haarpcache:

    mysql -u root -p
    grant all privileges on haarp.* to haarp_user@localhost identified by 'haarpcache_password';
  

  Change the username ('haarp_user') and password ('haarpcache_password') for which you want.

• Configure your /etc/haarp/haarp.conf

    CACHEDIR <dir_1>|<dir_2>|<dir_3> ...	
    MYSQL_USER <haarp_user>
    MYSQL_PASS <haarpcache_password>
  

  <dir_1>, <dir_2>,... are the directories where cache files will be stored, for default '/haarp/'. <haarp_user> and <haarpcache_password> are the user and password for the database of haarpcache defined above.

• Copy and paste the line above at the end of the file 'squid.conf' (or 'squid3.conf' which is in operation):

    acl haarp_lst url_regex -i "/etc/haarp/haarp.lst"
    never_direct allow haarp_lst
    cache deny haarp_lst
    cache_peer 127.0.0.1 parent 8080 0 proxy-only no-digest
    dead_peer_timeout 2 seconds
    cache_peer_access 127.0.0.1 allow haarp_lst
    cache_peer_access 127.0.0.1 deny all
  

• Below the line of the http_port configuration, add:

    acl google url_regex -i (googlevideo.com|www.youtube.com)
    acl iphone browser -i regexp (iPhone|iPad)
    acl BB browser -i regexp (BlackBerry|PlayBook)
    acl Winphone browser -i regexp (Windows.*Phone|Trident|IEMobile)
    acl Android browser -i regexp Android
    request_header_access User-Agent deny google !iphone !BB !Winphone !Android
    request_header_replace User-Agent Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  

  with this we avoid the redirects HTTP to HTTPS in youtube.com. Only works for PC's, and when you enter to http://www.youtube.com from the address bar (of your browser) or from www.google.com.

• On squid.conf Comment the line:

    #hierarchy_stoplist cgi-bin ?
  

• On crontab add the line:

    0 0     * * *   root    /etc/init.d/haarpclean
  

• View reports of your cache system:

    cp /usr/src/haarpcache/etc/haarp/haarp.php /var/www/
  

  Change the line 24 of the file haarp.php with your user and password for your database haarpcache.

     if (!( $db = new PDO('mysql:host=localhost;dbname=haarp', 'haarp_user','haarpcache_password') ) )
  

• Your viewer:

    http://<IP_Your_Server>/haarp.php
  

• Security:

    iptables -I INPUT 1 -i <eth_wan_1> -m tcp -p tcp --dport 8080 -m state --state NEW -j DROP
    iptables -I INPUT 1 -i <eth_wan_2> -m tcp -p tcp --dport 8080 -m state --state NEW -j DROP
    ...
  

  The port 8080 is of your haarpcache server. You can change this editing the file /etc/haarp/haarp.conf.

• Avoid the QUIC protocol connections on www.youtube.com

    iptables -A FORWARD -i <eth_lan> -p udp -m udp --dport 80 -j REJECT --reject-with icmp-port-unreachable
    iptables -A FORWARD -i <eth_lan> -p udp -m udp --dport 443 -j REJECT --reject-with icmp-port-unreachable
  

  add the following line in squid.conf:

    # Disable alternate protocols
    reply_header_access Alternate-Protocol deny all
  

• All iptables rules should be placed in a *.sh file to be executed whenever the computer is started.

• Finally:

    /etc/init.d/haarp restart
    squid -k reconfigure
  

• Handling QoS can be done by making a marking of packages based on the search for the string "X-Cache: HIT from HAARP" or "HAARP: HIT from" in the packet header. Thus a possible handling may carried out as follows:

    IF_LAN=eth0
    MAX_DOWN=1300kbps
    MIN_CACHE_DOWN=1000kbps
    MAX_CACHE_DOWN=1100kbps
     
    iptables -A OUTPUT -t mangle -o $IF_LAN -p tcp -m string --string "X-Cache: HIT from Haarp" --algo kmp -j MARK --set-mark 666
     
    tc qdisc add dev $IF_LAN root handle 1:0 htb default 10 r2q 15
    tc class add dev $IF_LAN parent 1:0 classid 1:1 htb rate $MAX_DOWN ceil $MAX_DOWN
    tc class add dev $IF_LAN parent 1:1 classid 1:66 htb rate $MIN_CACHE_DOWN ceil $MAX_CACHE_DOWN
    tc qdisc add dev $IF_LAN parent 1:66 handle 66:0 sfq perturb 30
    tc filter add dev $IF_LAN protocol ip parent 1:0 handle 666 fw classid 1:66
  

Reinstall or Update

For a common update:

	su
	cd /usr/src/
	rm -fr haarpcache
	killall haarpclean 2>/dev/null
	git clone https://github.com/keikurono/haarpcache.git
	cd haarpcache
	./configure
	# IF you have problems with the last command, then (for debian/ubuntu systems) do:
		apt-get install g++-4.4
		./configure CXX=g++-4.4
	#
	make
	make install
	mysql -u root -p < haarp.sql
	# ... and enter your password of Mysql.

	cp -b etc/haarp/haarp.lst /etc/haarp/haarp.lst
	/etc/init.d/haarp restart
	squid -k reconfigure

Monitoring Logs

Depending on the location of the logs defined in the file /etc/haarp/haarp.conf:

	tail -f /var/log/haarp/access.log
	tail -f /var/log/haarp/error.log

Level information in the logs: change the LOGLEVEL on /etc/haarp/haarp.conf.

List of Plugins

• Currently Haarp supports:

  Youtube (https not supported), Netflix (Video and Images), DoubleClick (Block of ADS), Googlevideo, Dailymotion, AOL, Vimeo, Metacafe, Modovideo, Bitgravity, Globo, Terra, Viddler, Justin, Edgecastcdn, Youku, Wrzuta, 5min, Viddler, Tumblr, Facebook (https not supported);

  Bomusica, Bullafina, Coqui, Cutmu, Domusica, Domusicas, Dungeonbusters, Goobeo, Gooveo, Miniclip, Molenillo, Mp3-Buscador, Musicaveo, Ning, Nofeehost, Sonicomusica, Turner, Vevos;

  4shared, Mediafire, Etrustdownloads, Ziddu;

  Akamaihd, Amazonaws, Blogspot, Imageshack, Issuu, Mercadolibre, Submanga, Ytimg;

  Avast, Avgate, Eset, Mcafee, Avira, Bitdefender;

  Windowsupdate, Adobe;

  Porntube, Xvideos, Pornhub, Redtube, YouPorn, Serviporno, Tube8;

  Appspot, Cloudfront, Juegosdiarios, Juegosjuegos, Netpocket, Nordeus, Popcap, Socialpointgames, Telaxo, Tetrisfb, Vostucdn, Wooga, Zgncdn;

  Disneylatino, Friv, Geewa, Hulkshare, Kixeye, Llnwd, Maguinamotors, Manabar;

  SpeedTest (some servers);

Configure your file /etc/haarp/haarp.lst to disable or enable the plugins.

Changelog

Version 2.1.2

2016/01/04

• New plugins: Nvidia (drivers), Blazzers (World Of Wordcraft, Diablo 3, etc), League of Legends.
• Update plugin akamaihd.
• Change of format in some columns of the databases: support for cache of files of great size.
• Disabled for default the general plugin (in BETA).

Version 2.1.1

2015/12/08

• Update of the eset plugin.
• Reducing the workload on the haarpclean.
• Limit of count of users in the database.

Version 2.1

2015/12/01

• Improve performance for the use of the Mysql.
• Fixed the problem: 'general protection ip:7fc4e8d35b9f sp:7fff604fde10 error:0 in libmysqlclient.so'.
• Update regular expression (in the haarp.lst file) for netflix, globo and amazonwas. Delete plugin for amazonwas domain.

Version 2.0b

2015/10/28

• Plugin general.
• Adding script for check status of haarp; update process installation from a script
• Update reg. expressions for multiples plugins; reedit some messages of errors for Debug mode.
• Fixed problem with reader of header HTTP.

Version 1.5.1

2015/09/26

• Fix problem: "segfault at 10 ip 000000000042ee66 sp 00007fffb74d2270 error 4 in haarp[400000+45000]".
• Manager of multiples ip in the header "X-forwarded-for".
• Update bash script of installation: force the redirect to haarp from squid in squid.conf.

Version 1.5

2015/09/14

• Storing the address IP of the user who uses the cache.
• Adding user history for each file in cache.
• Update the regexp for some plugins in haarp.lst.
• By default, the 75% of the hard disk is used.

Version 1.4.1 Thanks to Hernan Dario Nacimiento.

2015/07/31

• Update the plugins for edgecastcdn, socialpointgames and akamaihd.
• Update plugin of Youtube: not cached of videos on live.
• Using the parameter ‘clen’ of the url from youtube as file size.
• Update the HaarpClean program.
• Fixed the problem of bad replacement string in the SQL requests.
• Update the file of configuration for logrotate.
• Fix problem: ‘general protection ip:7f39b6cdd44e sp:7ffc278771f0 error:0 in libmysqlclient.so.18.0.0‘.

Version 1.4

2015/07/07

• Adding 50 new plugins.
• Improved support for DASH technology.
• Update the file of configuration for logrotate.
• Fix problem: 'segfault at 6 ip 00007f850c2cef36 sp 00007fff19f22d00 error 4 in libmysqlclient.so.18.1.0'.
• Fix problem: 'Error in /usr/local/sbin/haarp: free(): invalid pointer: 0x00007fb5e166d618'.

Version 1.3

2015/06/13

• Fix Problem with concurrent caching of files.
• Update the plugin Youtube.

Version 1.2.1 Thanks to Samuel Espinoza, Oscar Vaquero y Fernando Maniglia.

2015/05/29

• Fix concurrent caching of files.
• Update haarpclean script.
• Update some plugins: Youtube, Dailymotion, 4shared, Issuu, Avgate, Submanga, YourPorn, PornHub, Xvideos, PornTube, Tube8.
• Files of youtube are cached with the watch ID from the URL.

Version 1.2

2013/12/24

• Improved ad-blocking plugin for Youtube.
• Update plugins: globo.com, youtube.com.
• Differentiation of the header for the chrome browser.
• Update of haarpclean script for the cleaning the cache.
• Synchronous to the storage of the cache in an only file.

Version 1.1

2013/02/09

• Cache dynamic and intelligent.
• Update of plugins for the smart operation.
• Multidisc.
• Blocking advertising to Youtube.
• Cache of videos in html5 for Youtube.

Fork created by

Manolo Canales Cuba

[email protected]

http://perufw.wordpress.com

If you find this code useful, or wish to fund further development, you can use PayPal to donate to the HaarpCache project: