Search Results for "parallel"

Crawl4AI is a high-performance, AI‑ready web crawler tailored for LLM data ingestion and RAG pipelines. It supports adaptive crawling heuristics (stopping when enough info is gathered), structured markdown output, and high-speed parallel execution. Designed to operate at scale with optional Docker deployment and framework integrations.

Trafilatura is a Python package and command-line tool designed to gather text on the Web. It includes discovery, extraction and text-processing components. Its main applications are web crawling, downloads, scraping, and extraction of main texts, metadata and comments. It aims at staying handy and modular: no database is required, the output can be converted to various commonly used formats. Going from raw HTML to essential parts can alleviate many problems related to text quality, first by...

...Scrapyd is an application (typically run as a daemon) that listens to requests for spiders to run and spawns a process for each one. Scrapyd also runs multiple processes in parallel, allocating them in a fixed number of slots given by the max_proc and max_proc_per_cpu options, starting as many processes as possible to handle the load.

Manage node sets, node groups and execute commands on cluster nodes in parallel. Provides an event-based Python library to improve administration of large compute clusters or server farms. Command line tools: clush and nodeset included.

SOURCE CODE MOVED TO https://github.com/kowalpy/Robot-Framework-FTP-Library . NEW RELEASES WILL APPEAR ONLY AT GITHUB AND PYPI. This Python library makes it possible to test or use FTP server using Robot Framework keywords. Together with the library, there are also available project example and keywords documentation created by libdoc. This sourceforge webpage is linked from http://robotframework.org/#test-libraries as external library so it can be considered as the official Robot...

Are you maitaining many machines and bored with the tedious operation of logging to those machines one by one just for checking the machine's status? Or are you losing patient for your simple script that have to run command in those machines in sequence? The corunner is just for those senarios. It's designed to speed up execution in your machines. It can executes command in handreds of or thousands of machines concurrently. Provide it with your files and command and the machines' location,...