JtR Development

This section contains information useful for those who might wish to understand the internals of JtR, or develop new functionality.

If you're new to JtR as I was, you might find yourself with nice clean compiles that run 'out of the box' (thanks to the design), but no idea what's really going on in the compilation process. If you want to put the JtR code into an IDE like NetBeans to understand this software better by debugging it, you'll need to know more about the compilation process. Same goes if you want to extend jTR with your own formats/algorithms etc.

So..here's a really brief description of the compilation process

1. You choose your target architecture from the command line eg. macosx-x86-64 as part of your call to make.
2. Based on this selection, the Makefile then creates a link from the right architecture header (eg. x86_64.h) to arch.h. This file defines things like how long a WORD is on your machine etc. and is used throughout the software. Throughout the compilation process, CFLAGS, LDFLAGS and Assembler flags have been setup specific to your machine.
3. Compilation occurs:
   • Utility files are compiled (unshadow, unafs, unique, undrop, genmkvpwd, mkvcalcproba, calc_stat), with a file suffix based on your architecture eg. .exe, .com or nothing for *nix
   • john related objects are compiled, along with your architecture/option specific assembler file(s) eg. x86_64.S containing, amongst other things, DES and other optimised functions for your machine.

Armed with this information and the right debug settings (eg. -g), you'll now find it's not too difficult to put together your own IDE-friendly Makefile for debugging and understanding this beast in more detail.

Let's say you've built a new decryptor format file for JtR and you want to include it in the compilation. Here's a very high level guide on the changes you'll need to make.

In john.c

1. Add your external structure in the list towards the top of the file eg. extern struct fmt_main fmt_XXXX;
2. In the function john_register_all, register your new format eg. john_register_one(&fmt_XXXX);

In options.c

1. Update the list of valid format to include yours. Change the text literal –format=NAME and add yours in at the end eg. the end of the line goes from …/HDAA\n”; to /HDAA/XXXX\n”;

Don't forget to update your Makefile, probably just adding your format object to the JOHN_OBJS_MINIMAL target eg. insert fmt_XXXX.o somewhere in there (preferably the format list which comes first). Lastly, if you want to offer up your contribution to the JtR world, follow the instructions to make a patch against an original version.

The default JtR distributions include a Makefile that prepares binaries of john and the utilities based on an architecture you provide as a parameter to make. While this setup gets you going fast, it doesn't allow you to configure the compilation for debugging and development. There are many possible ways to do this setup, however one that's working well for me is described below in the hope others will find it useful. I use NetBeans 6.5 on an iMac, however the steps are adaptable to any IDE you care to use.

1. Create a new C/C++ project in NetBeans, call it john. We'll be using conditional compilation, and extra profiles to test different runtime environments.
2. Create a set of logical directories to hold the different categories of files in JtR. This is immensely helpful when you immerse yourself in development, and is instructive in and of itself for understanding the design of this software:
   1. Assembler Files
   2. Format File
   3. Header Files
   4. Key Files
   5. Password Files
   6. Resource Files (default project creates this)
   7. Source Files (as above)
   8. Utility Files

1. Now, allocate each of the base distribution files to one of the above categories…as follows:

Assembler Files - just choose your machine's architecture and pick the corresponding .S file. Don't add the other .S files - this is just done for simplicity. You can rename your architecture .h file to arch.h for simplicty if you want. In this situation, your makefile doesn't have to take your command line (from make clean XXXX) to work out what to link to - good for IDE's where modifying Makefiles (and making your changes stick!) is difficult.

   Format Files - add every *_fmt.c file here. 
   Header Files - the .h's
   Key Files - Any test key files you use. The ones included with the package are: xxx xxx
   Password Files - password.lst, and any other password files you create/get. 
   Source Files - any other .c files left over EXCEPT the utility standalones (next category)
   Utility Files - unshadow, unafs, and unique
   Character Files - defaults are all.chr, digits.chr, lanman.chr, alnum.chr and alpha.chr
   Important Files - just leave the default Makefile in here. We'll be customising for our architecture later, with some special make steps.

1. In your IDE's prebuild step, ensure you a) link your architecure file (eg. x86-64.S) to arch.h, and run sed (per the original makefile lines) to replace unsigned long with ARCH_WORD in a few files (this keeps the original files intact). You may also need to run other steps: again - check the original JtR Makefile for your architecture, and add them to the pre-build steps of your IDE's Makefile (this is called .pre-build in NetBeans).
2. Make sure you update your project settings with the correct compilation flags for your architecture - see the original Makefile. In particular, don't forget the .S assembler files too!
3. If your IDE does not allow the straightforward flagging of files you do not want compiled, you should remove the utilities .c files from your project. The reason for this is that they include a main function, and your final LD linker may complain about multiple definitions of main. These files are best.c, calc_stat.c, detect.c, genmkvpwd.c, mkvcalcproba.c, and symlink.c

You should now have a project that will product the john executable file, in your IDE, and hopefully, debuggable so you can learn more about the guts of this utility.

• The format load and run process and sequence eg. providing a format test, running format self tests, updating fmt_main with the steps that 'override' (it's C) the ones in format.c

• What goes in your default format functions like XXXX_get_key

• Running JtR efficiently eg. what run profiles to do when so you maximise the hits in the most efficient manner eg. run a wordlist first, then with rules etc