This project offers a Python scripting environment to build a general-purpose BitTorrent search engine. The engine monitors users currently active in one or more transfer activities on two large BitTorrent public networks. The Python scripting environment is in beta testing phase and is planned to be released in late 2020. Since the script is in Python, it is cross-platform (Windows, Linux, etc.) and can be used with a relational database engine to build a private torrent index site. Running btScript on 4 low-cost Linux computers can collect a one-million unique torrents in a matter of a few days.
BitTorrent has established itself as a robust, fast, and efficient message transfer protocol to transfer large files among peer-to-peer networks. In trackerless DHT networks, the use of trackers or special servers to locate the location of desired files is optional. There are two major popular trackerless networks in use, the mainline DHT and the Vuze DHT network. The mainline DHT uses the LibTorrent (LT) messaging protocol. The Vuze DHT network supports both the LibTorrent (LT) and the Azureus (AZ) messaging protocols. The mainline DHT network is much larger than the Vuze DHT network and can have as many as 200 million peers active at a moment. The Vuze DHT network is much smaller but still can have as many as 1 million users at a time.
Most BitTorrent clients support only the LibTorrent message protocol to enable peer-to-peer data transfer on the mainline DHT network. Only a few BitTorrent clients (Vuze, BiglyBT, Transmission) support both message protocols to access the mainline DHT and the Vuze DHT network at the same time. The Vuze message protocol allows the title of torrents in each peer potentially participating in transfer activity to be seen by any other peer in the network. The address of up to 50 active peers currently associated with a particular peer can also be seen. This capability makes possible features such as “Related Content” in the BitTorrent client Vuze and the automatic detection of new contents from peers participating in a BitTorrent swarm without having to rely on external torrent sites or any other type of search engine.
Due to a huge and diverse volume of digital contents in many languages available on the DHT networks, both public and copyrighted, it is desirable to have a general-purpose torrent search engine to find new and recent contents without having to rely on traditional torrent index sites, search engines or RSS feeds. The search engine has the ability to monitor specific peers without having to participate in any type of contents transfer activities. Over a certain period of time, finding or indexing new contents by monitoring the downloading queue of a specific group of active peers and thus requesting data transfer from these specific peers will render more relevant and satisfactory results. This should be a signficant feature of a torrent search engine because most top hits returned by most if not all traditional search engines based upon categories from index sites or feeds most often do not match the user interest or simply unavailable to download.
Vuze is the only BitTorrent client that offers the automatic discovery of new contents via its “Swarm Dicovery” and “Related Contents Management” feature. This feature makes Vuze unique among the many outstanding BitTorrent clients because it offers users a mechanism through which to find new content, one that does not involve visiting a torrent site. This mechanism finds contents to download based on the activities of other BitTorrent users that have downloaded the same or similar contents as you do. In other words, the “Swarm Discoveries” feature helps BitTorrent users to anonymously relate one piece of downloaded content with
Users have always been looking for simple ways to get more content that they might enjoy, new materials similar to the kind of contents they have already downloaded. Searching the traditional search engines or existing torrent discovery sites tend not to offer anything beyond a small set of predetermined categories such as by format or by genre. Most users are looking for something that works across many similar sites, while at the same time having the capability to focus in on things that may be of interest.
“Swarm Discoveries” is based on the concept of related contents. In a nutshell, if you currently downloads torrents A, B and C and there are two or more users in the swarm download torrents A, B, D and A, B, E, then there is a strong likelihood that you might be interested in torrents D and E as related contents as well. The amount of related contents will grow quickly since at any moment there are millions of people exchanging contents via the mainline and Vuze DHT network.
Since Vuze is open source software, its core design and plugin modules can be expanded and modified to suit our needs. By bringing this related content feature of the “Swarm Discoveries” to the next level by automating the detection and acquisition of related contents from the DHT network, a local database which acts like a private content discovery site can be built in just a matter of days.
btScript is a group of convenient Python functions designed to seamlessly integrate the Java and Python platform by exposing the application programming interface (API) of the open-source Azureus engine to the Python scripting environment. Developers no longer need to design, develop and test existing or added features to the Azureus engine on the Java platform.
All features of the Azureus engine including plugins can be accessed and developed strictly in the interactive Python scripting environment using btScript. This capability enables the rapid development and testing of new BitTorrent applications since the Azureus engine is already known as a stable, robust, and time-tested engine for BitTorrent applications.
The Azureus engine is the only platform that makes torrents on the visible Internet (clearnet) available on I2P and vice versa via its plugin architecture. If the user adds a torrent from I2P, it will be seeded on both I2P and the clearnet, and if a user adds a torrent from the clearnet, it will be seeded on both the clearnet and I2P. Therefore, torrents previously published only on I2P are now available to the entire visible Internet, and users of I2P can download any torrent on the Internet while maintaining the anonymity of I2P.