Problem with preg_match_all in phpcrawlerutils.class.php

Anonymous — Mon, 27 Jul 2009 21:39:59 -0000

In file: phpcrawlerutils.class.php 201
if there is strlen($source) >= 400000 preg_match_all produces a segmentation fault.

I solved the problem adding this:

if ( strlen($source) > 50000 ) {
$source = substr($source, 0, 50000);
}
preg_match_all("/<[ ]{0,}a[ \n\r][^<>]{0,}(?<= |\n|\r)(?:".$match_part.")[ \n\r]{0,}=[ \n\r]{0,}[\"|']{0,1}([^\"'>< ]{0,})[^<>]{0,}>((?:(?!<[ \n\r]*\/a[ \n\r]*>).)*)<[ \n\r]*\/a[ \n\r]*>/ is", $source, $regs);

You can contact me at: <a href="http://www.informaticaautonomos.com">http://www.informaticaautonomos.com</a>

Base HREF not being considered and incorrect links result

zoltak — Tue, 31 Jul 2007 07:10:09 -0000

If a document that contains a base href such as <base href="http://mysite.com/test/" />

With links such as <a href="test/test-things">Things to test</a>

The makeFullQualifiedURL function does not build out the URL correctly.

Refer to modified files for fix.

I added made the main program loop lookup the base href and in the makeFullQualifiedURL it detects if the baseHref exists and builds the URL as required.

The crawler now traverses a site with base hrefs :)

Recent changes to bugs

Problem with preg_match_all in phpcrawlerutils.class.php

Base HREF not being considered and incorrect links result