User manual page is not available anymore

Amalia Criznic — Wed, 13 Sep 2023 09:02:20 -0000

user manual page is not availalbe anymore https://web-harvest.sourceforge.net/usage.php

Saving list in a var-def

Adin Israel López Muñoz — Mon, 08 Mar 2021 21:57:10 -0000

I have the following code to retrieve data from a table ina webpage:

<?xml version="1.0" encoding="UTF-8"?>

<config>
    <var-def name="webpage">
        <html-to-xml>
            <http url="http://somepage.www/tablepage"/>
        </html-to-xml>
    </var-def> 
    <loop item="currPro">
        <list>
           <xpath expression="//tr/td">
             <var name="webpage"/>
           </xpath>     
        </list>
        <body>
            <var name="currPro"></var>
        </body>

    </loop>

</config>

I need to know how the loop logic should be inside to be able to save all the elements of the list in a variable var-def

I need to save all the elements of a list in a variable var-def

Adin Israel López Muñoz — Mon, 08 Mar 2021 21:53:19 -0000

I have the following code to retrieve data from a table ina webpage:

<?xml version="1.0" encoding="UTF-8"?>

<config>
    <var-def name="webpage">
        <html-to-xml>
            <http url="http://somepage.www/tablepage"/>
        </html-to-xml>
    </var-def> 
    <loop item="currPro">
        <list>
           <xpath expression="//tr/td">
             <var name="webpage"/>
           </xpath>     
        </list>
        <body>
            <var name="currPro"></var>
        </body>

    </loop>

</config>

I need to know how the loop logic should be inside to be able to save all the elements of the list in the same variable var-def

gives java heap space

Amalia Criznic — Sun, 10 Mar 2019 20:17:26 -0000

Hello Guys,

I have a simple request to a webService which returns an XML ( about 100mb) which is not that much from my point of view , and when try to clean the content by <html-to-xml> in order to do the xslt it stops and gives a java heap space error
I even set jvm -Xms750m -Xmx4048m but still the same </html-to-xml>

Do you have a solution for this ?

BR,
Amalia

Cannot web scrape this url

WEETAT — Mon, 14 Jan 2019 09:14:07 -0000

Hi all
I need to web crawling the page URL:
https://stomp.straitstimes.com/singapore-seen/moe-to-take-action-against-lewd-instagram-account-targeting-junior-college-girls

We have error in line of scraper.execute();

private Map<string, object=""> crawlArticleData(String sourceUrl, Resource configFile) throws IOException {</string,>

    InputSource configIn = new InputSource(configFile.getInputStream());
    //Execute Web Harvest process to extract content
    ScraperConfiguration config = new ScraperConfiguration(configIn);

    Scraper scraper = null;
    String articleContent = "";
    List<String> imageUrls=null;
    try {

        scraper = new Scraper(config, "");
        //Config timeout for httpClient used in Scraper
        org.apache.commons.httpclient.HttpClient scraperHttpClient = scraper.getHttpClientManager().getHttpClient();
        scraperHttpClient.getParams().setParameter("http.socket.timeout", new Integer(SO_TIMEOUT));
        scraperHttpClient.getParams().setParameter("http.connection.timeout", new Integer(CONNECTION_TIMEOUT));

        scraper.addVariableToContext("articleUrl", sourceUrl);
        scraper.execute();

        //get article content
        Variable articleContentVariable = scraper.getContext().getVar("articleContent");
        if (articleContentVariable == null || articleContentVariable.toBinary() == null) {
            logger.debug("Fail to extract body content for " + sourceUrl);
            return null;
        }

        articleContent = new String(articleContentVariable.toBinary(), "UTF-8");
        if ( !org.apache.commons.lang.StringUtils.isBlank( articleContent ) && articleContent.length() > 2000000 )
        {
            logger.info( "Reject article because content is too large. Article size: " + articleContent.length() );
            throw new IOException();
        }

        Variable imageUrlVariable = scraper.getContext().getVar("imageUrl");

        imageUrls = new ArrayList<String>();

        if (imageUrlVariable != null) {
            String imageUrlsString = imageUrlVariable.toString();
            if (!StringUtils.isEmpty(imageUrlsString)) {
                imageUrls.addAll(Arrays.asList(imageUrlsString.split("\n")));
            }
        }
    }catch (Exception  ex ){
          logger.error( "Crawler error sourceUrl="+sourceUrl, ex);
    }finally{
        //clean up
        scraper.dispose();
        configIn.getByteStream().close();
    }

    Map<String, Object> articleData = new HashMap<String, Object>();
    articleData.put("articleContent", articleContent);
    articleData.put("imageUrls", imageUrls);

    if ( !org.apache.commons.lang.StringUtils.isBlank( articleContent ) && articleContent.length() > 0 )
        logger.info( "Crawl completed. Article content size: " + articleContent.length() );
    else
        logger.info( "Crawl completed. " );

    return articleData;
}

Help parsing a javascript/JSON variable

Rick Goncalves — Mon, 01 Oct 2018 22:37:43 -0000

Hi,

I'm new to web-harvest, any help you can provide would be much appreciated.

I have a page that is outputting javascript containing a JS variable: domainList. This variable value is a JSON script, so I assume I can use JSON to XML to then xpath the items/name values. I'm looking to build a list of the name values.

here is the actual javascript text I need to parse. thanks in advance.

<script type="text/javascript">
             if (self != top) {
                 top.location = self.location;
             }
             require(["dp/layers/loginlayer","dojo/domReady!"],function(){
                require(["dojo/parser", "dp/Login", "dojo/ready"], function(parser, Login, ready){
                  ready(function(){

                    parser.parse();

                    var domainList = {
                        "identifier": "id",
                        "items": [
                        { "name": "default", "id": "default" }, { "name": "dp_Common", "id": "dp_Common" }, { "name": "hr_etime", "id": "hr_etime" }, { "name": "hr_us_ew2", "id": "hr_us_ew2" }, { "name": "isc_tools_mfg_omcs", "id": "isc_tools_mfg_omcs" }, { "name": "mq_Adapter", "id": "mq_Adapter" }, { "name": "partnerworld_websvc_bpw", "id": "partnerworld_websvc_bpw" }, { "name": "sc_prod", "id": "sc_prod" }
                        ]
                    };

                    login = new Login();
                    login.startup(domainList,"XI52",
                        "b03eixmlapp005",
                        "",
                        "",
                        "XI52.7.6.0.9");
                  });
                });
             });
            </script>

Invoke WebHarvest function from Java function

sunilprabakar — Tue, 07 Feb 2017 11:58:08 -0000

I have created web harvest function. I am able to invoke the function using web harvest code. My challange is, need to invoke that web harvest function from java function. Is it possible? For example considered this
We harvest method

 <function name="testing">
 <script><![CDATA[
   // code block
   ]]></script>

Java method

<script><![CDATA[
  function jMethod(){
    testing(); // need to call that web harvest method here
  }
  ]]></script>

Orchid Tor Proxy

parker20121 — Sat, 14 Jan 2017 17:15:45 -0000

TWIMC:

I'm trying to use WebHarvest with Orchid (https://subgraph.com/orchid/index.en.html), a Tor proxy.

I started Orchid on my local machine, and it's supposed to create a proxy at port 9150.

D:\Programs\webharvest>java -jar orchid-1.0.0.jar
Jan 14, 2017 11:58:52 AM com.subgraph.orchid.TorClient start
INFO: Starting Orchid (version: 1.0.0.8c6b26d)
Jan 14, 2017 11:58:52 AM com.subgraph.orchid.directory.DirectoryImpl loadFromStore
INFO: Loading cached network information from disk
Jan 14, 2017 11:58:52 AM com.subgraph.orchid.directory.DirectoryImpl loadFromStore
INFO: Loading certificates
Jan 14, 2017 11:58:52 AM com.subgraph.orchid.circuits.CircuitCreationTask checkCircuitsForCreation
INFO: Cannot build circuits because we don't have enough directory information
Jan 14, 2017 11:58:52 AM com.subgraph.orchid.directory.DirectoryImpl loadFromStore
INFO: Loading consensus
Jan 14, 2017 11:58:52 AM com.subgraph.orchid.directory.consensus.ConsensusDocumentImpl verifySingleAuthority
WARNING: Consensus signed by unrecognized directory authority: 0232af901c31a04ee9848595af9bb7620d4c5b2e
Jan 14, 2017 11:58:52 AM com.subgraph.orchid.directory.consensus.ConsensusDocumentImpl verifySingleAuthority
WARNING: Consensus signed by unrecognized directory authority: 23d15d965bc35114467363c165c4f724b64b4f66
Jan 14, 2017 11:58:52 AM com.subgraph.orchid.directory.DirectoryImpl loadFromStore
INFO: Loading microdescriptor cache
Jan 14, 2017 11:58:52 AM com.subgraph.orchid.directory.DirectoryImpl loadFromStore
INFO: loading state file
>>> [ 80% ]: Connecting to the Tor network
>>> [ 85% ]: Finished Handshake with first hop
>>> [ 90% ]: Establishing a Tor circuit
>>> [ 100% ]: Done
Tor is ready to go!

I tried adding the proxy settings through WebHarvest > Settings and at the command line with parameters proxyhost=127.0.0.1 proxyport=9050. When running the application, the connection doesn't seem to go through the proxy.

Any help would be greatly appreciated.

Regards,

Cannot include path=functions.xml

jomy nn — Fri, 15 Jul 2016 09:13:02 -0000

function.xml
Please download and extract this file
http://web-harvest.sourceforge.net/download/webharvest2b1-project.zip
or using this code

<?xml version="1.0" encoding="UTF-8"?>

<config>
<!--
Download multi-page list of items.

    @param pageUrl       - URL of starting page
    @param itemXPath     - XPath expression to obtain single item in the list
    @param nextXPath     - XPath expression to URL for the next page
    @param maxloops      - maximum number of pages downloaded

    @return list of all downloaded items
 -->
<function name="download-multipage-list">
    <return>
        <while condition="${pageUrl.toString().length() != 0}" maxloops="${maxloops}" index="i">
            <empty>
                <var-def name="content">
                    <html-to-xml>
                        <http url="${pageUrl}"/>
                    </html-to-xml>
                </var-def>

                <var-def name="nextLinkUrl">
                    <xpath expression="${nextXPath}">
                        <var name="content"/>
                    </xpath>
                </var-def>

                <var-def name="pageUrl">
                    <template>${sys.fullUrl(pageUrl.toString(), nextLinkUrl.toString())}</template>
                </var-def>
            </empty>

            <xpath expression="${itemXPath}">
                <var name="content"/>
            </xpath>
        </while>
    </return>
</function>

</config>

Latest Build for WH

Malay Shah — Thu, 14 Jul 2016 08:14:47 -0000

Hi Team,

Was having a scenario where I need to login(by sending POST) on a particular website and download a file, but the site uses re-direction(uses 302 return code).

When I tried using the WH 2.0 build from http://web-harvest.sourceforge.net/download.php, it wan't working, its returning the same login page as a response.

So, I tried checking out the Trunck code from https://sourceforge.net/p/web-harvest/code/HEAD/tree/trunk/ and tried creating a local build and there it executed correctly.

Just wanted to know, where can I find the latest build for the code on trunk? or Where are you planning to release the next build?

Regards,
Malay

Recent posts to Discussion