CHM readers for Linux

October 16th, 2008

The other day I needed a CHM reader. CHM files are Microsoft HTML compiled files and I knew getting a reader on Linux might be some what tricky.

After trying to open the file in Firefox (I thought I might get something after all it is ‘HTML’) and failing I realised that there was most likely a Firefox plugin for CHM functionality. Indeed there is, https://addons.mozilla.org/en-US/firefox/addon/3235

Automate Rapidshare free downloads - No more captchas

July 14th, 2008

Earlier this month the popular file sharing service Rapidshare decided to drop the use of captcha’s. This came as a huge surprise to myself because previously Rapidshare has used some of the most convaluded captcha puzzles ever! At points I would be Lucky to get 25% of captures used by Rapidshare right.

Dropping the use of captcha’s has paved the way for automated downloads but Rapidshare are aware of this and has lowered the download speed of free users to 500 kilobit per second.

Never less I would rather the lower speed and with the capability to have the damn thing automated. So here it is, my simple bash script to download a batch of files of Rapidshare. Hell if you only want to download 1 file its only a “one” linner.

URL=$(curl -s <URL TO RAPIDSHARE FILE>| grep "<form id=\"ff\" action=\"" | grep -o ‘http://[^"]*rar’);ourfile=$(curl -s -d "dl.start=Free" "$URL" | grep "document.dlf.action=" | grep -o ‘http://[^"]*rar’ | head -n 1); sleep 90;wget $ourfile;

Full version to download a batch of files specified in a file called input.txt, 1 file per line.

New Version (1/11/2008): Rapidshare has added a wait time in between file downloads. On top of your download
to start. This has been fixed.

Download the full .sh with some enhancments made by Tune. Also see comments for a Python version by Conman.

Download Itay’s version with some nice features. See his comment for the features.

while read line
do

URL=$(curl -s $line | grep "<form id=\"ff\" action=\"" | grep -o ‘http://[^"]*rar’);
ourfile=$(curl -s -d "dl.start=Free" "$URL" | grep "document.dlf.action=" | grep -o ‘http://[^"]*rar’ | head -n 1);
sleep 90; #70 secs for 200mb 50 secs for 90mb
wget $ourfile;

done < input.txt

Analysis

while read line
do
Code……………..
done < input.txt

Loop through all lines in the file input.txt, store each line in a variable called ‘line’, move to the next line.

URL=$(curl -s $line | grep "<form id=\"ff\" action=\"" | grep -o ‘http://[^"]*rar’);

Use curl to ‘download’ the page of the url we are processing. Pipe the downloaded pages source code through some greps to extract the action URL of the html form which we need to ’submit’ to trigger the download. Store this url in a variable called ‘URL’.

ourfile=$(curl -s -d "dl.start=Free" "$URL" | grep "document.dlf.action=" | grep -o ‘http://[^"]*rar’ | head -n 1);

Using curl again, ’submit’ a html form to our extracted URL, passing it the post data (-d switch in curl) of being a free user. Rapidshare then replies with a new page with a list of Rapidshare servers to download from, they are stored in lines of Javascript starting with ‘document.dlf.action=’. After the equal sign is the direct url to our file, so lets grab this with another grep!

We are now have a list all the Rapidshare servers holding our file like this.

http://rs202l3.rapidshare.com/files/50141870/1554876/centos-5.0.tar.part01.rar
http://rs202l33.rapidshare.com/files/50141870/1554876/centos-5.0.tar.part01.rar
http://rs202tl3.rapidshare.com/files/50141870/1554876/centos-5.0.tar.part01.rar
http://rs202cg.rapidshare.com/files/50141870/1554876/centos-5.0.tar.part01.rar
http://rs202l32.rapidshare.com/files/50141870/1554876/centos-5.0.tar.part01.rar
etc…..

We will try our luck with the first one, so filter this output through ‘head’ telling it to extract one line (head -n 1)

sleep 90;

Rapidshare still makes us wait for our file! So, we will just wait here for however long it takes which is 70 secs for a 200mb (new file size limit) 50 secs for 90mb (old file size limit).

wget $ourfile;

Finally the wait is over, luckily its the script doing the waiting, not us. Lets get our file. wget our file that is…

Closing comments, if you know how to improve the script please let me know. At the moment it will only work with .rar files though you could easily change this. A generic version would be nice but this is what most files are in on RS.

A version which uses proxys to download files in parallel would be good.

I dont like piping a grep command into a grep command so I’m sure this could be done with just one grep.

I use use both wget and curl, which here accentually do the same job, if you have one or the other you can fix the script up to only use either.

Tip: Set your batch download up an in a screen session and detach it so you can ssh in and monitor your downloads where ever you go. This will also let your downloads continue downloading when you close your terminal window or putty sesion.

Programs needed: Bash (duh!), head, grep, sleep and curl or wget

Note: I have required to increase the sleep time to 90 seconds. I am unsure how to calculate this time because it has seemed to change from when I first wrote this. It may well be RS has increased the time or the value is variable depending on how much you have previously downloaded. So 90 seconds seems like a safe value for now.

Gemapis - Milestone 1 tech demo

May 28th, 2008

This project was started over a year ago, though 95% of the work has been done in the last few weekends (to be honest it seems like I spent more time on the video and blog post then on it itself).

Gemapis is a map/tile editor for 2D tile based games such as RPG’s and Platformer’s. Typically it will be used for games resembling the graphics of SNES or Sega Megadrive (Genisis) era of games.

 I have released a video showing off Gemapis’s features which were todo’s in milestone one which is now complete.  Theres a few big things to do in milestone two before a first release.

 

Features you saw in the video include:  

  • Tile layers
  • Tile transparency
  • Tile rotation
  • Tile mirroring
  • Group tile selection
  • Saving the tile data and reloading it
  • Saving map data into a binary format which will be able to be read by many systems and programming languages.

Features I will/would like to include:

  • Undo/Redo of map edits
  • Nice user interface
  • Polygon hit detection (expressed in XML)
  • Map entity specifications Eg, Sprites, NPC’s, Events etc (expressed in XML)

Like I said in the video. If your interested in helping let me know; you will get SVN access (like with most of my projects).  I ask for help because I cant be good at everything ;)

Gemapis will be open source software.

 In the tech video/demo for milestone 2 you will see a C++ game using a map created in Gemapis and hopefully using all the features described above.

High resolution video can be found here.

 

Your-V-Lyrics

May 3rd, 2008

Heres a Web 2.0 mashup I made in a few hours in PHP. 

The idea of Your-V-Lyrics is to search for a artist and a song so you can watch/listen to the video clip as well follow the lyrics next to the video.

So far it works quite well. Especially for music videos where the director really embraces the lyrics and theme of the song.

 Try it out!. Here are some of my favourite music videos

Deftones - Change

System of a down - Arials

Tool - Schism

Metallica - Frantic

Rammstein - Ich will

Incase you wanted to know, videos are from youtube.com and lyrics from lyricwiki.org

Status update

March 1st, 2008

It’s been a while between blog posts partly because I’ve been busy; partly because I have been lazy.

I will discuses 3 topics here so if one doesn’t interest you don’t read it.

Firstly I want to talk about my new laptop and Windows Vista.
My old Compaq P3 laptop died a few months back (closer to a year back) I was surprised how much mistreatment it actually survived through but it eventually chocked with my wrong doing so I its death is on my shoulders.
Ever after that tragic day I have been looking for a new one, even my pc was getting close to being due for an upgrade with slower compile times, Flash videos using nearly 100% CPU and barely only being able to play 4 year old games at low frames.

So I bought a Asus M51SN, the version I purchased has a Core Duo 2.5Ghz CPU 6MB cache(L2), 2 GB ram, 200GB HDD, 512MB 9500M GS GFX (Geforce 9), TV Tuner and Vista Ultimate. It also came with a free wireless router.
The hardware is very capable and the CPU and GFX architecture were only released last month.

Setting up my free Asus wireless router was kind of tricky. It came with very basic lame instructions which required you to run a setup cd on Windows and press a button on the device to put it into a ‘find me setup’ mode. I can see why there is so many un-secured wireless networks when these things are designed to be easy to setup but this makes them defautly un-secure.

Using some assumptions and guessing I was able to setup the router the right way without having to use the crappy 2 step instructions in the manual. Knowing that routers by default have the class C IP address of 192.168.1.1 I was able to hit the router setup page from my browser. After a few attempts I was then able to guess the default administrator username and password (which was ‘admin’ and ‘admin’) and login to configure the wireless encryption password and some other junk. The default username and password wasn’t even documented in the manual.

I was having some initial problems accessing the rest of the network and the internet because the setup was….
MyWirelesssRouter—-MainRouter—-Internet/RestOfHomeNetwork

After trying allot of settings I realised its probably because MyWirelesssRouter and MainRouter probably have the same IP address. This was the case, upon changing MyWirelesssRouter’s IP from 192.168.1.1 everything worked. The thing I hate about networking compared to programming is that there is no debug mode. You usually have to use some kind of logic to work your problems out. Obviously things like ping, traceroute and nmap work in allot of situations but not all.

The most surprising thing about my laptop is I actually like Vista. This may be because I had such low expectations with everything I had heard. From a general users point of view I would say it’s better than XP! From an technical point of view I will tell you it’s memory foot print is allot larger and its install size is fucking huge. Vista Ultimate also comes with Windows media Center which I really like also.
Speaking of Media Center, At the moment I’m Writing this in Word 2007 (which at the moment my opinion is simply ‘its different’ ) while watching Gyroscope on JTV on ABC2. Which I can pause and rewind. Im also told ABC2 has Red Dwarf which I can’t wait to watch also.

The only 2 issues I have run across so far is there is no port of Nvidias Nview application for Vista which isn’t MS’s fault. The other main issue is there is a bug where my DVD drive will stop working some times. Once I reboot all is fine with it.

The main thing I don’t like about Vista is well… its Windows :p

Need antivirus, endless OS updates, proprietary, inferior kernel design and basically everything else detailed in the OS comparison section of Eric Raymond’s book ‘The Art of Unix programming’

Next Im going to talk about a bug that I came across on code I maintain but didn’t write.

The bug only ever happens on one day every 4 years. Yes that’s right it’s a leap year bug! The code was essentially doing age validation, it would take the current day, month and year then minus x amount of years from the current year. It would then check to see if your age was less than the max age limit (current date – x years).

Well you see the date on this particular day was the 29th of February 2008 (last Friday).
So the code was something like

DateTime.parse(29,02,2008 - x);

Where x is 2. The problem is 29th of February 2006 (2008-2) doesn’t exist! So when it goes to parse the date it throws an invalid date exception! Making problems even worse is this genius of a programmer put this code in a try catch block and would handle the exception of a invalid date with the error message along the lines off ‘you are too old’ which was also the error you got if in fact you were to old.

This is only one error which plagues the whole project. How could of this been easily avoided? Use the DateTime library to do the hard work, library writers are always nearly 1000 times smarter than you and their code has been tested that many times better also. A much more logical way to write this would have been

DateTime.parse(29,02, 2008).minusYears(2);

Lessons to learn: Do not use incorrect error messages and generic exceptions, use library’s when they are available and most importantly from my quotes page ’Always program as if the person who will be maintaining your program is a violent psychopath that knows where you live’

Finally, A quick review on games I’ve been playing.

Hitman Blood money PC, nice graphics and physic. Doesn’t really bring much new to the game play from the previous Hitman titles.

Pokemon Perl (DS), Dosnt offer much new game play from previous Pokemon games, graphics are quite bad for a DS game, allot of the sound is still taken from the original gameboy game, doesn’t make use of touch pad much. I look forward to playing it online though.

Final Fantasy 3 remake (DS), great game in all aspects. Starts of a little slow but gets allot better. Only downside is it lacks mini games which would have been made so totally awesome with the features of DS (duel screen, touch pad and microphone).

Netsend!

December 26th, 2007

I just want to talk about a small application Ive been working on and a few other things.

For very small periods on and off over the last 6 months I have been working an application I am calling netsend. Netsend is / will be a networking tool with the main aim of sending a file across a network peer to peer. Weather is be across a LAN or the internet.

Netsend is person to person. Peer to peer (single peer). The Idea is simple, I want to send Mr-x a picture of a purple monkey. I simply run netsend. Mr-x connects and the transfer starts. After the transfer is complete both programs on mine and Mr-x machines end. In contrast it will also be able to compress a bunch of pictures of purple monkeys, encrypt them and send them to a backup server every night.

Yes I know theres FTP, secure copy, chat protocol x or some other similar application.

But netsend differs in that its not a service like FTP, its only function is to successfully copying 1 file at a time. Its not designed just for humans sending files like with chat clients (although you can get a interactive command line). It can be 100% automated. It will be 100% open source and multi platform.

The idea of netsend is like any good OS/Networking tool which allot of the unix paradigm is designed around. Make it do one job and do it well, use IPC (Inter Process Communication) and use standard Input and Output to make it generic. Also lots of options and features helps also but no feature creep.
At the moment its features are short and platforms are limited.

  • Percentage completion status bar
  • File resume
  • Windows 32

Here are some screen shots with a transfer through the loopback network.
But I have had the exact same success across the internet and a LAN.

First screen capture is of a transfer taking place. 2nd is of an md5 checksum of the original file and then of the sent file. The checksum is the same, the transfer went 100%

Netsend p2p network transfer Netsend p2p network transfer
If you are interested in trying a very very early release of netsend download it here. Sorry Im working on getting it ported to more systems (Linux is next). Note the executable is pretty large because it hasn’t been stripped of debugging information or optimized. Netsend will be designed to be light weight and require very few external library’s.

The next feature I want to add is specifying a range or single IP which can make a connection.

The other thing I wanted to talk about was Microsoft’s File Checksum Integrity Verifier.

Seems like a pretty good free command line utility to check that byte by byte bit by bit two files are the same. You can generate checksums in MD5 or SHA1. I will be keeping this handy.

Your programmer personality type

November 17th, 2007

Very long time between posts. Ive been really busy.

Just a quite post about a interesting online programmer personality test I found. My results can be found below.
Your programmer personality type is:

DLSB

You’re a Doer.
You are very quick at getting tasks done. You believe the outcome is the most important part of a task and the faster you can reach that outcome the better. After all, time is money.

You like coding at a Low level.
You’re from the old school of programming and believe that you should have an intimate relationship with the computer. You don’t mind juggling registers around and spending hours getting a 5% performance increase in an algorithm.

You work best in a Solo situation.
The best way to program is by yourself. There’s no communication problems, you know every part of the code allowing you to write the best programs possible.

You are a liBeral programmer.
Programming is a complex task and you should use white space and comments as freely as possible to help simplify the task. We’re not writing on paper anymore so we can take up as much room as we need.

Just some quick thoughts on my results - Seems Im a Doer rather than a Planner. This is probably true in some essence. I wont plan small to medium projects but this is probably because im a Solo programmer rather than a Team programmer so I know in my head how I want something to be done anyway. Though I do believe on large projects with a large team just like testing you can never do enough planning.

I think Im a Solo programmer because I program to get away from people. The computer does what I say. No questions. If things screw up its my fault. Stupid people who needs em… I find programming in a Team is only good if its a really tight team. Though im really all for concepts like Open Source.

Seems Im a Low level programmer. I tend to disagree with this. I will more likely only go to a low level language when the problem requires/suits it though I am more likely to choose JAVA or .net. It does annoy me how much JAVA and VB do abstract you from the hardware sometimes you just want to do pointer arithmetic or make use of unsigned data types. C# is a bit better that VB or JAVA in that respect.

It probably thinks Im Low level because I answered ‘Allow me to make use of system resources’ in the following question.

The ‘perfect’ language will:

I answered this because I dont think a language that allows anyone to program is practical. Computers are complex and I wish people would stop pretending otherwise. Languages like JAVA are THAT EASY that anyone who passed high school maths should be able to do simple programming tasks in it now anyway.

Well, thats all for now. See you all in a few months.

ABreakoutC - Another break out clone

August 26th, 2007

Today I have finished the first release of a Breakout clone I have been working on for PC (Linux / Windows / PC).

If you dont know what Breakout is; it was one of the first arcade games ever made. It involves hitting a ball at blocks using a paddle which you can move left and right. Breaking all the blocks progresses you to the next level. Like latter versions of breakout, ABreakoutC includes power-ups which add special effects to the game.

ABreakoutC was my first game written in c++ using SDL which is a cross platform game/media library.

It and its source are released under the GPL3 and the graphics were made in entirely in GIMP.

The game was developed in Linux with the help of the Netbeans (best IDE in the world) c++ plugin.

Game Objects

abreakoutcitems.png

Controls

Pause/Resume - P
Start the ball moving - Space bar
Move paddle - Left key, right key

Screen shots

abreakoutc1.png abreakoutc2.png abreakoutc3.png

Downloads

Windows
x86Linux-32
Source

System requirments

1Ghz or better Celeron. 128 Meg or ram.
Note - You cant go higher than 60fps

If you want to design your own levels (let me know and I will talk you through it), provide music to the game or even add a power-up your welcome to. Getting a few people working on this would be great. Also I would like someone to do a Visual c++ build as mine is only MinGW.

———Update!———

If you have previous downloaded any version of the game please get version 002 (from the above locations) there was a bug which will cause what seems like a random crash on Windows which is now fixed.

For those that are interested it was a memory violation caused by…
if(ds[i]->blockDestroySprite1->status == Sprite::FINISHED){
delete ds[i]->blockDestroySprite1;
ds.erase(ds.begin() + i);
}

gs->apply_surface(ds[i]->location_X,ds[i]->location_Y,
ds[i]->blockDestroySprite1->spritesheet,ds[i]->blockDestroySprite1->getFrame());

As you can see I was deleting the sprite then trying to draw the sprite on certain instances. I have no idea why this error didnt occur on Linux (where I did all of my testing). I have now fixed all versions to delete the sprite after it is drawn.

It makes very little sense but Im not going to complain.

August 6th, 2007

Well today started out great.

Today was a ‘Bank holiday’ for NSW Banks and because I work in the Insurance industry I guess I got the day of as well. It makes very little sense but Im not going to complain.

All was going good until some time around 4:00pm I decided to run a program I hadnt worked on in a few weeks. The problem was it didnt work and not only did it not work it wouldnt go away and sat on 100% CPU.

So anyway I killed it. But the problem wasnt it, it was some other process it had spawned which was now running at 100% and owned by root.

So I logged on as root and not even thinking of the effects I kill -9ed it. At this point the kernel pretty much crashed. I had to hard reboot.

BIOS loaded. Grub tried to load. But failed. It failed with the Error: ‘error #17 cannot mount selected partition’. Great. Anyway, I pulled out my now trusty Live Distro SAM Linux and booted it. I went in to inspect my partitions all was fine. WTF? I inspected my grub.config. No issue there.

Then the magic happed. I shut down my PC (properly this time) and took out SAM and rebooted.

All was fine. It makes very little sense but Im not going to complain.

SAM Linux - Review

July 7th, 2007

SAM Linux Live Distro.

Ok lets begin from the start. I insert the Live CD and boot. I come to the boot menu which gives me several options including to boot SAM Linux, boot into console mode, memory tests and a few other options.

Login user names and passwords are shown on the boot menu for both a normal user and root user which is good because on of the first live cds I tried I could never find the root password. Its great there is a boot to console mode, it cuts down on memory and loads a little faster. Also half the reason you want a live CD is so you can quickly go and repair a setting in VI (Emacs isn’t included) or backup a some files to a FTP server or ssh account or even a removable drive. So you can then format the non bootable Windows PC and reinstall an an OS again (hopefully Linux this time ;) but more on these things latter.

One thing that was disappointing is the network configuration Wizard isn’t included if you boot to a console only environment which means depending on your network type and network interface getting a connection could mean allot of configuration.

Also a nice feature was the ability to hide and show the kernel loading and the systems boot process by simply toggling it on and off with the escape key. Ive used some distros that try to be more user friendly by hiding this information but when something goes wrong like a drive doesn’t mount your left wondering why.

Once your logged in and the intro music plays your left with the Xfce Windows manager. Which I like to think is a light weight Gnome. I say this because Xfce 4 is based of the Gnome tool kit 2 (Gtk2). Xfce is perfect for a live cd because its not only light but a very powerful desktop manager.

Network

Before you get to the desktop you can opt to set up a network adapter through a convenient wizard. The wizard interestingly had options for GPRS, Edge and 3G which is I believe are wireless cell phone transport technologys. There were also for options for IDSN, modem, Ethernet, DSL and even Blue tooth! I also have a good feeling that wireless connections wouldnt be to difficult to get running provided you had the correct Windows driver.

Network tools

SAM comes with several networking tools which above others includes Firefox 2, gFTP, ssh (must have for me)/ Putty (which seems to be emulated through WINE), TightVNC, Skype, Gaim (must have for me), Xchat and Gnutella client (I would of preferred a Bittorrent client).

Multimedia

I dont really expect much multimedia capability’s to be on a live CD but SAM comes packed full of options. My media player of choice is Mplayer which as a bonus comes with a nice GUI. I was able to watch every multimedia file on the PC including a xvid/divx encoded movie and play a WMA encoded song with no extra codecs required, I would be interested to know if commercial DVD playback works straight out of the box as well and wasnt able to check this out as the live cd occupied my only drive. As for Mp3 players both Winamp(XMMS) and Itunes (Exaile) clones/alternatives are included.

Development / programming

This is a interesting section. Im in two minds weather you actually need development tools on a live CD. But as myself being a programmer I’m going to say its a good thing and here’s why.

Perl - Yes Perl is included, Perl is an amazing powerful OS scripting language and can be used to automate a complex task.

GCC - (Gnu Compiler Collection) I guess this could come in handy if you need to do some quick hacking.

Python - I think this was included primarily for Xchat. But hey its there! I even use a few Python programs my self Shuffle-db http://shuffle-db.sourceforge.net so I cant complain.

IDE - Disappointingly there is no good IDE and not even a editor with syntax highlighting included. On top of this the GNU debugger (GDB) isnt even included. Having said that, there is ‘Bluefish’ which seems to be a good web development IDE.

Windows support

As I mentioned earlier one thing I use a live cd for is to resolve issues with a computer which is either miss-configured or non booting or even to get important files before the final solution of a system wipe.

SAM Linux makes this ideal with its many networking applications and support for Windows programs and partitions.

Wine (the Windows emulator (sic)) is included which allows you to run handy Win32 system applications such as Regedit.exe to get rid of that malicious software or fix miss configured setting.

It also comes with fat32 read/write and NTFS read support.

Eye candy.

If you wish SAM Linux gives you the option of turning on Beryl and Compiz which gives you desktop effects that out amaze even that seen in MS Vista. Whats even move amazing is I didnt even have to install any ATI graphics drivers for my 9600 pro GFX card. This allowed me to play the 3D OpenGL game ‘Warsow’ which I had on another partition straight out of the box!

Speaking of FPS games - Tilda is also included, which is a Terminal emulator which works like the consoles in First Person Shooter games. So when you press the ‘~’ key a terminal scrolls down where you can execute bash commands and pressing ‘~’ again will send it scrolling back out of the way until you need it again.

Other notable programs

JAVA, The full Open Office 2 suit (which im using now), Gimp, Gnome Baker (CD/DVD burning) a PDF viewer and a archiving tool. Even the Flash player is installed.

Disk/Partition management

SAM Linux also comes with an Disk partitioning tool which can handle many types of file system which is always useful on a live cd.

System/Software management

System management is basically identical to Mandriva’s. Which I have found to be probably the best out of any distros.

Software management takes place in Synaptic which is a computer user interface and front-end to the Advanced Packaging Tool. I am still unsure how difficult it would be to get packaged software for PCLinux (which same Linux) hopefully you would be able to use the Mandriva packages because PCLinux is based of Mandriva. If this is not the case finding aditonal software would be a problem until the distribution becomes more popular.

Final comments

SAM Linux is a very exceptional live CD and would make and very nice light Linux OS if installed.

The inclusion of ‘commercial’ software such as flash and 3D GFX card drivers make the experience even better. SAM linux dosnt excel in any one area which makes it a hugely versatile portable OS.

I could install SAM to my HDD right now and not need any more additional software packages apart from GDB and a good IDE but for 90% of uses they could use it straight out of the box.

The combination of Xfce and Beryl and Compiz make for a smooth experience on the front end yet SAM includes enough versatility behind the scenes to get the ‘real’ work done when needed.