!!!OLD!!! Contributed patches for John the Ripper

Incrementals for 1.7.9 Jumbo-4

Patch Author Status and Description
0001-External-mode-Truncate-at-format-s-max-length.patch magnum Ensures that external mode truncates candidates to format's max length.
0002-NT-and-NT2-No-length-check-in-set_key.patch magnum If patch 0001 is accepted, we can drop some checks in very fast formats for a boost, like NT (NT2 already lacked them). The patch also makes the UTF-8 length checks (which are still needed) in NT and NT2 faster (and accurate))
0003-update-x86-ssei.h-to-match-jumbo-4-x86-sse.h.patch magnum Replicates the jumbo-4 changes in x86-ssei.h to x86-ssei.h
0004-SSE_PARA-fixes.patch magnum More granularity (different versions of gcc, or using clang) to the SSE_PARA choices in arch.h. icc was not affected so no rebuild needed of .S files. Also fixes hmacmd5 so it handles para > 5 and changes icc target for a boost in both build time and performance.
0005-mssql05-new-faster-set_key.patch magnum The new SSE set_key() is put in the mssql05 format.
0006-mssql05-length-fix.patch magnum Length fix for mssql05.
0007-mssql05-regression-fixes-for-generic-builds.patch magnum Hopefully last regression fixes for mssql05
0008-dynamic-non-contiguious-number-issues_BE.diff JimF Fix for BE systems which have removed a couple of dynamic formats, so the list (built in), is not contiguious. Also fixes some formatting issues in dynamic from the jumbo, and uses ARCH_INDEX macro, instead of a cast to unsigned char

Jumbo rebase for 1.7.9

Patch Author Status and Description
0001-john-1.7.9-jumbo-1-1.7.8-jumbo-8-equivalent.patch.gz magnum This is roughly equivalent of 1.7.8-jumbo8. Apply to a core 1.7.9 tree. The dynamic format will be broken after this patch, until you apply 0002.
0002-Fixes-for-dynamic_fmt-vs.-new-MD5_std.c.patch magnum This fixes dynamic format for new MD5_std.c
0003-All-30-patches-originally-for-1.7.8-jumbo-8.patch.gz magnum This is supposed to exactly match the 30 incremental patches originally for 1.7.8-jumbo-8.
0004-All-salt_hash-functions-use-params.h-macros-for-size.patch magnum Fixes salt_hash() for all Jumbo formats (best effort, I believe it's OK but some future tweaking may be called for)
0005-Implement-new-binary_hash-get_hash-sizes-for-a-bunch.patch magnum Implements new binary hash sizes for a bunch of formats. Note: In an early versions of this patch, dynamic_fmt also got the new sizes. But they both fail (on Pix formats, only) on Sparc and the largest one fails on intel. Getting #5 to work should be trivial but #6 would need to look at next word.
0006-Sparc-fix-for-sapB.patch magnum Sparc (alignment) fix for sapB. Turns out this was likely needed in Jumbo-7 already.
0007-sse-intrinsics.c-typo-used-MD5-macro-in-MD4-block.patch magnum Corrects a typo in sse-intrinsics.c
0008-x86-ssei.h-update-to-match-x86-sse.h.patch magnum Rebase x86-ssei.h on new x86-sse.h
0009-Re-add-undrop-to-john.c-after-it-was-lost-during-reb.patch magnum undrop was lost in translation
0010-Update-relbench.pl-to-latest-version-by-Solar.patch magnum Replace relbench.pl with latest version by Solar
0011-vc-cygwin-fixes.diff JimF Patch needed for cygwin-x86-any to run without core dumping, and fix for VC build
0012-md5-sse2i-strlen-bug.diff JimF MD5 was crashing on SSE2i builds. This was due to the buffer not being initialized like it was (now it is dynamic alloc). This was causing a strlen() to walk off the end of the buffer
0013-Bartavelle-s-mdfourmmx-added.patch magnum Adds Bartavelle's asm functions for mmx/sse2 MD4 as posted here. I had to add a pusha at top and a popa at bottom in order to get it working. That was just a guess after looking at mdfivemmx, there are possibly some minor fixes needed (preferably by someone who actually knows assembler). Having said that, this works like a champ on linux 32-bit builds.
0014-raw-md4-tweaks-new-faster-set_key.patch magnum Adds mmx/sse asm support for Raw-MD4, but more importantly boosts the intrinsics a lot too by using a new highly optimised “32-bit” set_key() insired by NT_fmt. The secret seems to be to avoid GETPOS.
0015-Alternate-NT-format-using-Bartavelle-s-functions.patch magnum This adds an experimental, alternative NT format using Bartavelle's mdfourmmx or intrinsics. It is currently a tad slower than Alain's NT format on most builds but it seems it beats it on sse2i. I'm not really wanting to replace Alain's format, this is mostly meant as a benchmark. Maybe we should put it in unused/ for reference?
0016-raw-MD5-thick-50-faster-than-dynamic.patch magnum This adds a temporary, thick, raw-md5 format that is 50% faster than dynamic_0 (depending on build of course) because of a highly optimised set_key(). I'm hoping we can use this just as a benchmark, and get dynamic_fmt up to similar speeds. That day, we'll just ditch this thick one (again!)
0017-md4-sse-patch-VC.diff JimF This patch corrects issues with VC builds, on magnum's new-modified formats. Also adds the VC marshalling functions to md4-mmx.S and md4.h, so that it can be built properly to be used by VC for mmx and sse2, just like MD5 and SHA1.
This provides portability fixes to patches 0013, 0014, 0015, 0016. Some of those patches tried to add VC, port, but was not done correctly.
0018-VC-more-patches.diff JimF 2nd Patch for VC compatibility. VC builds is a joint task using MinGW32 to build some of the asm stuff (x86.o, md5(md4,sha1)-mmx.S, x86-mmx.S and x86-sse.S). One change in patch 0017 was a problem. Also removed many #ifdef _MSC_VER, within the arch.h files
0019-Dynamic-alloc-of-MKPC-arrays-for-mskrb5-net-lm-mscha.patch magnum Dynamic memory allocation for the hungriest formats. Reduces BSS size by an order of magnitude. Discussion.
0020-Raw-SHA-SHA0-added.patch magnum Vanilla SHA0 format, using OpenSSL.
0021-nsldap-set_key-speedup.patch magnum Implements the faster vector version of set_key() in NSLDAP.
0022-A-couple-more-dump_stuff-functions-for-debugging.patch magnum A couple debugging functions.
0023-Dynamic-alloc-other-fmts.diff JimF Dynamic alloc for 3 other formats (cash1, crc, XSHA512). Those were the others listed in Solars email about .bss getting out of hand in OMP builds.
0024-Dynamic-alloc-fmt-memset-patches.diff JimF This patch memsets the allocated blocks to NULL bytes, to 'simulate' static buffer allocation. Many formats were failing without this, because of making assumptions that the buffer starts out with null byte data.
0025-nsldap-fix-for-non-simd-builds.patch magnum Regression fix for nsldap.
0026-Dynamic-calloc-fmt-statics.diff JimF Rework of the .bss to dynamic memory alloc in the larger formats (patches 0019 to 0024). A new function created in memory.c mem_calloc_tiny() which works like mem_alloc_tiny, but performs a memset to null bytes. Also fixed bugs (alignment bugs), in patch 0023
0027-crc32-cap-OMP-num-threads-at-4-and-dynamically-alloc.patch magnum Dynamically allocate crc32 buffers for actual number of threads, and cap its number of threads at 4 (does not scale past that anyway). Also includes fixes to bench.c so it reports the capping.
0028-mscash1-dynamic-alloc.patch magnum mscash1, only allocate buffer size that is actually used.
0029-nsldap-regression-and-performance-fixes.patch magnum Hopefully the last regression fixes for nsldap… but Windows builds may need tweaking
0030-rawmd5-thick-simplification-optimizations.diff JimF Changed raw-md5-thin from 48 byte pw's to 55 byte.
Changed raw-md5-thick to have 55 byte pw (from 54 bytes). Reduced most differences between PARA_MD5 and non-PARA. In non-PARA, we pre-set the length, and then call the function that does not set length. This was about a 3% speedup (32 bit md5-mmx.S builds). Using this as a test case to reduce differences between the intrinsic and non-intrinsic. In the end, there should be little or no differences.
0031-hmac-md5-SSE2-fixes-2.diff JimF hmac-md5 was broken for SSE2, since the change magnum did switching from get-salt to salt for the processing. The 'total_len' var was not being set any more. Now, we have again removed 'some' of the PARA vs non-PARA changes. We set the length when building the salt, and then re-pull that length right before doing the 2nd MD5, thus getting proper length.
0032-build-change-fixing-core-issues-in-cygwin.diff JimF cygwin is causing strange cores (I can not explain them), in john. I have had to have john.c compiled with -O0 for a while. It causes no problems anywhere, since optimizations in the john.o do not impact any 'real' performance.
0033-nsldap-tweaks.diff JimF Fix for MSVC new 'SWAP' macro. BSWAP is available on any 486 CPU or better, magnum had it requiring SSE2. Also switched many unsigned int* to be ARCH_WORD_32* pointers. Also simplified the hashing functions.
0034-NT2-format-truncation-bugfix-for-non-SSE-builds.patch magnum NT2 format, truncation fix for non-SSE builds
0035-dynamic-fast-keyloading-and-7hashes.diff JimF 2 changes to dynamic. First, faster keyloading, for raw-md5 type formats, md5($p), md5(md5($p)), etc, and raw sha1. Code taken from changes to nsldap and raw-md5thick. Also, a new intrinsic swapping function added (bswap instruction on many Intel platforms).
2nd change is going from 5 to 7 hashing functions, which improves performance for LARGE numbers of candidates. There were some issues on pix format, since each limb in pix did not have enough bits to work properly on the 7th hash level, so these formats never get to that level. This gets the patch 0004-hash-size working properly for this format.
0036-nsldap-hash-table-sizes-and-use-of-johnswap.h.patch magnum New hash table sizes for nsldap
0037-salted-sha1-new-faster-set_key.patch magnum New faster set_key() for salted-sha1 format
0038-NT2-more-profiling-tweaks.patch magnum NT2 format, more profiling tweaks. I just can't get it as fast as NT except for 32-bit sse2i builds (but that one is 24% faster!). It is not really a fair fight though, as NT is reversing steps while intrinsics does not. Oh, and this patch also renames NT2_fmt(…) to nt2_fmt so the old format gets higher precedence.
0039-crypt-MD5-allocate-SSE-buffers-dynamically.patch magnum Crypt-MD5, dynamic allocation of the SSE out buffer too. And drop the imposed limit on max number of OMP threads.
0040-mscash2-dynamic-allocation.patch magnum mscash2 (DCC2), dynamic allocation of buffers, and drop the imposed limit on max number of OMP threads.
0041-Make-debug-functions-accept-any-data-type.patch magnum SSE debug functions tweaks
0042-mscash2-enable-OMP-for-generic-builds.patch magnum Enable OMP for generic builds of mscash2
0043-dynamic-salt-rewrite.diff JimF Extension and recoding of the dynamic format's salt handling. Now, all salt data ownership, and comparisions are done within the format. Also, the format now 'normalizes' the salt data to an internal format. This is due to no constraints being put on the salt format, thus the same salt data could be written in multiple ways. Also included is the HEX$ code written form 1.7.8 jumbo by Didier Arenzana. The code in this patch is now run within the salt() function, and not the get_salt() function. Also, within valid, we do some salt length validations. This now properly takes into account HEX$ fields. Also the HEX$ fields, are done in salt, salt2, username, and all F[x] fields, IF they are present. The format structure name, which was still md5-gen, has been renamed dynamic.

This code reduces the memory footprint to most salted dynamics some (since the format is smart enough to only store the right amount of data). Also, loading of many salted hashes is much faster, and runtime can be up to a few percentage faster. All 'hard' logic for salting has been moved into the salt() function. The set_salt() function now only sets pointers and 'lengths'. It is much quicker.
The data which john stores as a salt from dynamic, is simply a pointer. It will always be 4 or 8 bytes (32 bit build, 64 bit build).
0044-Didier-Arenzana-s-radius2john.pl-and-dynamic-formats.patch Didier Arenzana
(added here by magnum)
Didier Arenzana's radius2john.pl and dynamic formats. See Using John to crack RADIUS shared secrets and announcement.
0045-Bugfix-for-dynamic_fmt-after-0043.patch magnum Bugfix for dynamic_fmt after 0043. Thin formats' salt pointers ended up sized as the SALT_SIZE they reported. This made PHPS segfault on 64-bit.
0046-johnswap-fix.diff JimF Patch to johnswap.h, for systems which are not linux, VC, or gcc 4.3+. Also was a double define in dynamic.c for this situation
0047-MSSQL05-BE-fix.diff JimF MSSql 05 was broken on BE systems, for 'encoded' data (utf8, cp1252, etc). This removes LE/BE differences, and now it works. Not sure why this was there. Code was left, and simply commented out.
0048-Dynamic-HEX-and-other-fixes.diff JimF $HEX$ fully validated, for all salt values, and has length of salt correctly processed.
dynamic_27/28 commented out on BE builds.
Format fixed, so that the preload array does not have to be contigious (dyna_29 valid, but 27/28 are not on BE systems).
Removed a few warnings, here and there in the format.
0049-NTv2-BE-port.diff JimF the new NTv2 format was broken on BE systems. The patch is ugly, but #defines should only cause this code to be compiled only if being built on a Big Endian machine

Numbered Patches after john-1.7.8-jumbo-8, which are intended to make jumbo-9 (or later jumbo)

NOTE, these patches are designed to be installed, in numeric order. To get to the bleeding edge latest version of john, start at john-1.7.8-jumbo8, and apply patches 0001-* 0002-*, etc, following any specific instructions, until all patches have been applied. At times, a patch may be 'optional', and if so, should be listed as such, within the information written about the patch.

Patch Author Status and Description
0001-j8_dynamic_spacechanges.diff JimF Status: added 2011-11-09. Removed the spaced from the 'dynamic-preload' strings, and not in code. Code reverted back to original form.
0002-j8-Corrects-truncation-in-dynamic-fmt-for-any-builds.patch magnum Status: added 2011-11-09. Fixes a truncation problem in dynamic format for non-SSE/MMX builds with UTF-8 input. With this patch, jumbo-8 passes all of Test Suite for linux-x86-32-any target.
0003-j8-sapG-use-signed-int-in-loop-variable-for-OpenMP-2.patch magnum Status: added 2011-11-09. Trivial fix for allowing building with older OMP versions (discussion)
0004-j8-Intrinsics-usage-tweaks.patch magnum Status: added 2011-11-09. Adds a 32-bit version of the pre-compiled intrinsics code and enables its use for all sse2i targets. Discussion. This patch also implements this related change for crypt-MD5
0005-j8-sse-intrisics-asm-win32-fixes.diff JimF Status: added 2011-11-10. Perl script added to 'fix' the 32-bit pre-compiled intrinsics code to be built under Win32 Cygwin (and hopefully MinGW). A new perl script in the run dir, and some Makefile tweaks.
NOTE after the patch, run this command: chmod +x run/sse2i_winfix.pl
0006-j8-Added-missing-BF-in-c3_fmt-subformat-list.patch magnum Status: added 2011-11-10. Cosmetic, adds BF to the list of subformats for generic crypt
0007-j8-Further-revisions-of-pre-built-intrinsics.patch magnum Status: added 2011-11-11. Unified pre-built intrinsics
0008-j8-fixes-for-relbench-for-non-realtime-systems.diff JimF Status: added 2011-11-11. For systems which only print a single time (vs printing real: ss virtual ss), the new relbench.pl did not parse properly. That is now fixed. It works on my Win32 builds at least.
0009-j8-intrinsics-fixes.diff.gz JimF Status: added 2011-11-11. Another update to the intrinsic .S code. This one names sse-intrinsic.S into sse-intrinsic-64.S, so that dependency on sse-intrinsic.o will always use sse-intrinsic.c and sse-intrinsic-64.o will always use sse-intrinsic-64.S
0010-j8-pix-md5-thin.diff JimF Status: added 2011-11-11. pix-md5 using 'thin' format into $dynamic_19$ to allow SSE intrisics to be utilized.
0011-j8-pot-prepare-null-string-bug.diff JimF Status: added 2011-11-11. The prepare call at .pot load time, required some null char* bulletproofing.
0012-j8-intrinsics-for-mssql-formats.patch magnum Status: added 2011-11-12. Adds intrinsics for MSSQL and MSSQL05 for a nice boost.
0013-j8-refuse-to-parse-badly-formed-NETNTLMv2-as-v1.patch magnum Status: added 2011-11-12. Refuse to load badly formed NTLMv2 hashes as LMv1 or NTLMv1
0014-j8-revert-bad-XSHA512-fix-from-0011.patch magnum Status: added 2011-11-12. Revert bogus fix for XSHA512 in patch 11. A proper test was already in there, and the added one broke it.
0015-j8-Fixes-bogus-use-of-SHA1_N_STR-in-a-couple-formats.patch magnum Status: added 2011-11-12. Fix cosmetical bugs in three formats, discussion
0016-j8-Added-a-bunch-of-new-experimental-make-targets.patch magnum Status: added 2011-11-12. Adds a bunch of new make targets for intrinsics and/or pre-built dito, completely untested.
0017-j8-intrinsics-for-mysql-sha1.patch magnum Status: added 2011-11-14. Adds intrinsics for mysql-SHA1.
0018-j8-Drop-another-unused-variable-from-dynamic_n.patch magnum Status: added 2011-11-14. Drop an unused variable from dynamic format, getting rid of compiler warnings.
0019-j8-mssql-formats-regression-fix.patch magnum Status: added 2011-11-14. Regression fix for mssql formats.
0020-j8-Unify-algorithm-name-MMX-SSE2-SSE2i.patch magnum Status: added 2011-11-14. Unify “algorithm” names (MMX, SSE2, SSE2i) so it's clear and consistent.
0021-j8-update-x86-ssei.h-to-match-x86-sse.h.patch magnum Status: added 2011-11-14. Updates x86-ssei.h so it matches jumbo-8 x86-sse.h (cosmetic, and only affects AVX/XOP)
0022-j8-Shorten-Makefile-target-list.patch magnum Status: added 2011-11-15. Reduces Makefile target list with 10 lines by using the format 'linux-x86-64[i]' where ”[i] is an optional letter for pre-built intrinsics”. The output is still 78 lines though :)
0023-j8-New-NSLDAP-format.patch magnum Status: added 2011-11-15. Replaces NSLDAP_fmt with nsldap_fmt which is based on raw-sha1 and thus has intrinsics. Moves old NSLDAP format as well as NSLDAPS (ssha) and OPENLDAPS (openssha) to unused (the two latter are replaced with the salted-sha format).
0024-j8-Add-dynamic_26-tag-to-raw-sha1.patch magnum Status: added 2011-11-15. Adds a $dynamic_26$ tag to raw-sha1 format. Discussion. This tag was already selected for the dynamic (a.k.a md5_gen) version of this format.
0025-j8-Correct-Cygwin-align-bug-workaround.patch magnum Status: added 2011-11-16. Corrects a couple of Cygwin align bug workarounds that I previously screwed up due to a misunderstanding of the problem.
0026-j8-sse2-intrisic-multiblock-md45-1.patch JimF Status: added 2011-11-16. Fixes multi block MD4/5 in sse-intrisics.c NOTE, MD4 was not tested (but did not break existing use in raw-md4). MD5 has been (in hmacMD5 format).
0027-j8-hmacMD5-SSE2i.patch JimF Status: added 2011-11-16. Adds SSE2i to hmacMD5 format.
NOTE, this depends on patch 26, and for “pre-built intrinsic” builds, you also need patch 28.
0028-j8-Pre-built-intrinsics-.S-file-rebuilt-for-patch-26.patch magnum Status: added 2011-11-16. Pre-built intrinsics .S files rebuilt for patch 26.
0029-j8-64-bit-fixes-for-hmac-md5.patch magnum Status: added 2011-11-16. 64-bit fixes for hmac-md5.
0030-j8-hmac-md5-25-speedup.patch magnum Status: added 2011-11-17. Another 25% speedup for hmac-md5. Valgrind profiling showed that we were spending ages in set_salt(), so the nasty parts was moved to get_salt() where it belongs.
Also includes regression fixes (non-MMX/SSE failed to build after patch 27).

NOTE, 'current' patches are found at Contributed patches for John the Ripper These are 'old' patches. Kept here for history. These were released on the Wiki, and when outdated by v2 (or replacement) diffs, the old stuff should be dropped here.

External links to patches:
Patch Author Description
phpass-3 patch for JimF Latest phpass/phpBB
Fast rawMD5 SSE2 patch for JimF Latest 'fast' rawMD5
Performance patch for JimF Several performance enhancements. Memory file, additional larger hash tables, less file IO, and others Big performance gains for the fastest of the algorithms (like raw-MD5). REQUIRES phpass-3 and raw-MD5-SSE2-1 patches to be install prior to this patch properly installing, due to patch collision issues.
MinGW build JimF MinGW32 / MSYS building patch. NOTE only install this patch, OR the VC patch below. Do not install BOTH of them. For MinGW, either patch will work. For VC, the VC patch is required.
Patch for M$ VC JimF Startings of a Visual Studio build method (only for x86-any at this time) This is a 'superset' of the MinGW patch. It will also require installing and using MinGW to to the initial 'make' VC will link in some .o files MinGW assembles from the *.S files. I will get a set of VC project files uploaded soon.
Salt+md5 patch for build Pete NOTE: Obsolete: use Jim's MD5 patches below, as they cater for this format in an improved, generic way. Adds an md5 format with a known salt as the first two characters. This format is often used in PHP eg. oscommerce (hence the name), or Magento (TM irubin LLC). Note it also includes md5_eq.c and md5_go.c and md5_go.h from the the jumbo patch, so just remove these if you are patching the jumbo version. The patch can easily handle salts of size other than 2 - although you will have to modify code before runtime to do this. Ideally, it would be dynamic and transparent.
Generic MD5 module for JimF (v2 Sept 08, 2009) Adds a GENERIC MD5 module to john. It handles 'generic' types of MD5 hashes, such as md5($p) md5(%p.%s) md5(md5($p).$s) etc. Right now, only 'built-in' types are handled, but code hooks are there to allow for custom entries to be added by john users (without having to open up and edit code or re-compile john). Patching Prerequisites: phpass-3, raw-MD5-SSE2-1 and the 'several performance updates' patches to be install prior to this patch properly installing due to patch collision issues. ( v2 gets md5-gen working with -test, adds -subformat= fixes some bugs and improves performance, adds phpass to generic, as md5_gen(17) )
Test data for Generic MD5 JimF (v2, Sept 08, 2009) Test data, and pw.dic file containing all passwords for the test cases. There is a test case for raw-md5, and for md5_gen(0) to md5_gen(15). Each when run through john should find all 1320 passwords. The salted hashes only have 66 unique passwords (but multiple hashes). The non-salted hashes use all 1320 passwords out of the password list file ( v2 test data added md5_gen(17) phpass, and 'native' phpass format )
Numerous option additions and some bug fixes for JimF Adds a lot of new options (and fixes a couple bugs. Options are: -show=left (shows uncracked hashes) -pot=potfile (use a file other than john.pot) -rules=Section -single=Section (specify the section, i.e. no need to rename the section in john.ini all the time) Added a 'max' to -salts -nodups (ignores dupe check logic) -field-separator-char=c (allows setting a field sep char, other than : ) Added ETA to time output, to better 'know' when the crack will be done. Fixed an invalid percentage listing shown in a couple areas (very large dictionary file, and in -inc modes) Removed some debug flags added by the 'performance patch' Patching Prerequisites: phpass-3, raw-MD5-SSE2-1 and the 'several performance updates' patches to be install prior to this patch properly installing due to patch collision issues. NOTE there IS a patch collision within options.c and the generic md5 patch No way around that. One or the other will not patch right, and the person patching will need to hand edit the options.rej file
mpi10 patch for 1.7.5 magnum This is just john- after hand-editing the rejects against 1.7.5. It is obsoleted by the fullmpi patch.

Back to John the Ripper user community resources.

john/patches_old.txt · Last modified: 2011/12/18 01:07 by magnum
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Noncommercial-Share Alike 3.0 Unported
Recent changes RSS feed Donate to DokuWiki Powered by PHP Valid XHTML 1.0 Valid CSS Driven by DokuWiki Powered by OpenVZ Powered by Openwall GNU/*/Linux