I have SMF and Mambo 4.6.0 working correctly. When I install JSMF RC1, I get error when patching (from red link below):

"Failed: Code to be patched not found in file: require_once( \'includes/frontend.php\' );/home/mydom/public_html/test//index.php"

The details of install setting are:

Forum URL:     
SMF Absolute Path: /home/mydom/public_html/test/forum    
SMF Table Prefix: smf_    
installed    SMF Forum installed on the same database as Joomla.
click to patch    Patch Joomla Source Files.
click to patch    Patch SMF Source Files (Note: Unpatch before upgrading SMF or installing mod.)
done [verify]    Optional: Synchronize Joomla and SMF users(username or real name).

How can I fix this?

It's only going to work with Mambo 4.5.4.  I worked with 4.6.0 and 4.6.1 quite a bit and could not get JSMF to run.  With 4.5.4 it will run with one small modification.

modification to the patch done to the index.php. 

comment the session_start(); called in the index.php.  SMF will start a session.