Archive for the 'Networking' Category

Automate Rapidshare free downloads - No more captchas

Monday, July 14th, 2008

Earlier this month the popular file sharing service Rapidshare decided to drop the use of captcha’s. This came as a huge surprise to myself because previously Rapidshare has used some of the most convaluded captcha puzzles ever! At points I would be Lucky to get 25% of captures used by Rapidshare right.

Dropping the use of captcha’s has paved the way for automated downloads but Rapidshare are aware of this and has lowered the download speed of free users to 500 kilobit per second.

Never less I would rather the lower speed and with the capability to have the damn thing automated. So here it is, my simple bash script to download a batch of files of Rapidshare. Hell if you only want to download 1 file its only a “one” linner.

URL=$(curl -s <URL TO RAPIDSHARE FILE>| grep "<form id=\"ff\" action=\"" | grep -o ‘http://[^"]*rar’);ourfile=$(curl -s -d "dl.start=Free" "$URL" | grep "document.dlf.action=" | grep -o ‘http://[^"]*rar’ | head -n 1); sleep 90;wget $ourfile;

Full version to download a batch of files specified in a file called input.txt, 1 file per line.

New Version (1/11/2008): Rapidshare has added a wait time in between file downloads. On top of your download
to start. This has been fixed.

Download the full .sh with some enhancments made by Tune. Also see comments for a Python version by Conman.

Download Itay’s version with some nice features. See his comment for the features.

while read line
do

URL=$(curl -s $line | grep "<form id=\"ff\" action=\"" | grep -o ‘http://[^"]*rar’);
ourfile=$(curl -s -d "dl.start=Free" "$URL" | grep "document.dlf.action=" | grep -o ‘http://[^"]*rar’ | head -n 1);
sleep 90; #70 secs for 200mb 50 secs for 90mb
wget $ourfile;

done < input.txt

Analysis

while read line
do
Code……………..
done < input.txt

Loop through all lines in the file input.txt, store each line in a variable called ‘line’, move to the next line.

URL=$(curl -s $line | grep "<form id=\"ff\" action=\"" | grep -o ‘http://[^"]*rar’);

Use curl to ‘download’ the page of the url we are processing. Pipe the downloaded pages source code through some greps to extract the action URL of the html form which we need to ’submit’ to trigger the download. Store this url in a variable called ‘URL’.

ourfile=$(curl -s -d "dl.start=Free" "$URL" | grep "document.dlf.action=" | grep -o ‘http://[^"]*rar’ | head -n 1);

Using curl again, ’submit’ a html form to our extracted URL, passing it the post data (-d switch in curl) of being a free user. Rapidshare then replies with a new page with a list of Rapidshare servers to download from, they are stored in lines of Javascript starting with ‘document.dlf.action=’. After the equal sign is the direct url to our file, so lets grab this with another grep!

We are now have a list all the Rapidshare servers holding our file like this.

http://rs202l3.rapidshare.com/files/50141870/1554876/centos-5.0.tar.part01.rar
http://rs202l33.rapidshare.com/files/50141870/1554876/centos-5.0.tar.part01.rar
http://rs202tl3.rapidshare.com/files/50141870/1554876/centos-5.0.tar.part01.rar
http://rs202cg.rapidshare.com/files/50141870/1554876/centos-5.0.tar.part01.rar
http://rs202l32.rapidshare.com/files/50141870/1554876/centos-5.0.tar.part01.rar
etc…..

We will try our luck with the first one, so filter this output through ‘head’ telling it to extract one line (head -n 1)

sleep 90;

Rapidshare still makes us wait for our file! So, we will just wait here for however long it takes which is 70 secs for a 200mb (new file size limit) 50 secs for 90mb (old file size limit).

wget $ourfile;

Finally the wait is over, luckily its the script doing the waiting, not us. Lets get our file. wget our file that is…

Closing comments, if you know how to improve the script please let me know. At the moment it will only work with .rar files though you could easily change this. A generic version would be nice but this is what most files are in on RS.

A version which uses proxys to download files in parallel would be good.

I dont like piping a grep command into a grep command so I’m sure this could be done with just one grep.

I use use both wget and curl, which here accentually do the same job, if you have one or the other you can fix the script up to only use either.

Tip: Set your batch download up an in a screen session and detach it so you can ssh in and monitor your downloads where ever you go. This will also let your downloads continue downloading when you close your terminal window or putty sesion.

Programs needed: Bash (duh!), head, grep, sleep and curl or wget

Note: I have required to increase the sleep time to 90 seconds. I am unsure how to calculate this time because it has seemed to change from when I first wrote this. It may well be RS has increased the time or the value is variable depending on how much you have previously downloaded. So 90 seconds seems like a safe value for now.

Status update

Saturday, March 1st, 2008

It’s been a while between blog posts partly because I’ve been busy; partly because I have been lazy.

I will discuses 3 topics here so if one doesn’t interest you don’t read it.

Firstly I want to talk about my new laptop and Windows Vista.
My old Compaq P3 laptop died a few months back (closer to a year back) I was surprised how much mistreatment it actually survived through but it eventually chocked with my wrong doing so I its death is on my shoulders.
Ever after that tragic day I have been looking for a new one, even my pc was getting close to being due for an upgrade with slower compile times, Flash videos using nearly 100% CPU and barely only being able to play 4 year old games at low frames.

So I bought a Asus M51SN, the version I purchased has a Core Duo 2.5Ghz CPU 6MB cache(L2), 2 GB ram, 200GB HDD, 512MB 9500M GS GFX (Geforce 9), TV Tuner and Vista Ultimate. It also came with a free wireless router.
The hardware is very capable and the CPU and GFX architecture were only released last month.

Setting up my free Asus wireless router was kind of tricky. It came with very basic lame instructions which required you to run a setup cd on Windows and press a button on the device to put it into a ‘find me setup’ mode. I can see why there is so many un-secured wireless networks when these things are designed to be easy to setup but this makes them defautly un-secure.

Using some assumptions and guessing I was able to setup the router the right way without having to use the crappy 2 step instructions in the manual. Knowing that routers by default have the class C IP address of 192.168.1.1 I was able to hit the router setup page from my browser. After a few attempts I was then able to guess the default administrator username and password (which was ‘admin’ and ‘admin’) and login to configure the wireless encryption password and some other junk. The default username and password wasn’t even documented in the manual.

I was having some initial problems accessing the rest of the network and the internet because the setup was….
MyWirelesssRouter—-MainRouter—-Internet/RestOfHomeNetwork

After trying allot of settings I realised its probably because MyWirelesssRouter and MainRouter probably have the same IP address. This was the case, upon changing MyWirelesssRouter’s IP from 192.168.1.1 everything worked. The thing I hate about networking compared to programming is that there is no debug mode. You usually have to use some kind of logic to work your problems out. Obviously things like ping, traceroute and nmap work in allot of situations but not all.

The most surprising thing about my laptop is I actually like Vista. This may be because I had such low expectations with everything I had heard. From a general users point of view I would say it’s better than XP! From an technical point of view I will tell you it’s memory foot print is allot larger and its install size is fucking huge. Vista Ultimate also comes with Windows media Center which I really like also.
Speaking of Media Center, At the moment I’m Writing this in Word 2007 (which at the moment my opinion is simply ‘its different’ ) while watching Gyroscope on JTV on ABC2. Which I can pause and rewind. Im also told ABC2 has Red Dwarf which I can’t wait to watch also.

The only 2 issues I have run across so far is there is no port of Nvidias Nview application for Vista which isn’t MS’s fault. The other main issue is there is a bug where my DVD drive will stop working some times. Once I reboot all is fine with it.

The main thing I don’t like about Vista is well… its Windows :p

Need antivirus, endless OS updates, proprietary, inferior kernel design and basically everything else detailed in the OS comparison section of Eric Raymond’s book ‘The Art of Unix programming’

Next Im going to talk about a bug that I came across on code I maintain but didn’t write.

The bug only ever happens on one day every 4 years. Yes that’s right it’s a leap year bug! The code was essentially doing age validation, it would take the current day, month and year then minus x amount of years from the current year. It would then check to see if your age was less than the max age limit (current date – x years).

Well you see the date on this particular day was the 29th of February 2008 (last Friday).
So the code was something like

DateTime.parse(29,02,2008 - x);

Where x is 2. The problem is 29th of February 2006 (2008-2) doesn’t exist! So when it goes to parse the date it throws an invalid date exception! Making problems even worse is this genius of a programmer put this code in a try catch block and would handle the exception of a invalid date with the error message along the lines off ‘you are too old’ which was also the error you got if in fact you were to old.

This is only one error which plagues the whole project. How could of this been easily avoided? Use the DateTime library to do the hard work, library writers are always nearly 1000 times smarter than you and their code has been tested that many times better also. A much more logical way to write this would have been

DateTime.parse(29,02, 2008).minusYears(2);

Lessons to learn: Do not use incorrect error messages and generic exceptions, use library’s when they are available and most importantly from my quotes page ’Always program as if the person who will be maintaining your program is a violent psychopath that knows where you live’

Finally, A quick review on games I’ve been playing.

Hitman Blood money PC, nice graphics and physic. Doesn’t really bring much new to the game play from the previous Hitman titles.

Pokemon Perl (DS), Dosnt offer much new game play from previous Pokemon games, graphics are quite bad for a DS game, allot of the sound is still taken from the original gameboy game, doesn’t make use of touch pad much. I look forward to playing it online though.

Final Fantasy 3 remake (DS), great game in all aspects. Starts of a little slow but gets allot better. Only downside is it lacks mini games which would have been made so totally awesome with the features of DS (duel screen, touch pad and microphone).

Netsend!

Wednesday, December 26th, 2007

I just want to talk about a small application Ive been working on and a few other things.

For very small periods on and off over the last 6 months I have been working an application I am calling netsend. Netsend is / will be a networking tool with the main aim of sending a file across a network peer to peer. Weather is be across a LAN or the internet.

Netsend is person to person. Peer to peer (single peer). The Idea is simple, I want to send Mr-x a picture of a purple monkey. I simply run netsend. Mr-x connects and the transfer starts. After the transfer is complete both programs on mine and Mr-x machines end. In contrast it will also be able to compress a bunch of pictures of purple monkeys, encrypt them and send them to a backup server every night.

Yes I know theres FTP, secure copy, chat protocol x or some other similar application.

But netsend differs in that its not a service like FTP, its only function is to successfully copying 1 file at a time. Its not designed just for humans sending files like with chat clients (although you can get a interactive command line). It can be 100% automated. It will be 100% open source and multi platform.

The idea of netsend is like any good OS/Networking tool which allot of the unix paradigm is designed around. Make it do one job and do it well, use IPC (Inter Process Communication) and use standard Input and Output to make it generic. Also lots of options and features helps also but no feature creep.
At the moment its features are short and platforms are limited.

  • Percentage completion status bar
  • File resume
  • Windows 32

Here are some screen shots with a transfer through the loopback network.
But I have had the exact same success across the internet and a LAN.

First screen capture is of a transfer taking place. 2nd is of an md5 checksum of the original file and then of the sent file. The checksum is the same, the transfer went 100%

Netsend p2p network transfer Netsend p2p network transfer
If you are interested in trying a very very early release of netsend download it here. Sorry Im working on getting it ported to more systems (Linux is next). Note the executable is pretty large because it hasn’t been stripped of debugging information or optimized. Netsend will be designed to be light weight and require very few external library’s.

The next feature I want to add is specifying a range or single IP which can make a connection.

The other thing I wanted to talk about was Microsoft’s File Checksum Integrity Verifier.

Seems like a pretty good free command line utility to check that byte by byte bit by bit two files are the same. You can generate checksums in MD5 or SHA1. I will be keeping this handy.