Broadcatch

From MLDonkey

Jump to: navigation, search

Broadcatching is the automation of queuing of files, usually BitTorrents files. For idea, complaints, or feature requests discuss it on the talk page or Email me.

Contents

[edit] Background

About a half year ago I was look for a way to broadcatch my anime. At the time Azureus was slow, memory hungry, and the web GUI just sucked. I being working on a bash script to automatically check and queue torrent files into mldonkey. This is the third version the of script (in continual beta), it can handles various formats of RSS files.

[edit] Installation

It probably is a good idea to place these files into its own directory. The files debug.log, rssdata, and cast.log will be created on run.

[edit] Requirements

  • bash?
  • wget
  • sed
  • xsltproc
  • mldonkey (with the web interface enabled)

[edit] Files

broadcatch.sh

The script, still is development phase so there's still debugging function active. The mlnethost var will need configure to provide communicate to the mldonkey server. 

#!/bin/sh
# Script based on http://linc.homeunix.org:8080/scripts/bashpodder
# Last revision 10/15/2006 
# Released Dec 31, 2006
# Release Notes: as you can see I've been testing it for quite a
#  while, so it should be fairly stable; config name has changed and 
#  debugging is much less verbose only reporting matches.  As always
#  you can look to see what exactly was changed between releases. 
#  And
#                      A Happy New Year to Everyone!

# User Vars
# Address to contact the web GUI of mldonkey with username and password
MLHOST=username:password@localhost:4080
CONFG="bc.conf"

# Debug Log (set to /dev/null to turn off)
DEBUG="debug.log"

# Make script crontab friendly:
cd $(dirname $0)
echo -e "\nExecuting $0 on $(date)" >> $DEBUG

# feed dump reset
rm -f rssdata

# Read the bp.conf file and wget any url not already in the catch.log file:
while read subscription
     do
     xmldata=$(wget $(echo "$subscription" | sed 's/[^@]*@\(.*\)/\1/') -q -O -)
     expression=$(echo "$subscription" | sed 's/\([^@]*\)@.*/\1/')
     
     # If $expression is blank or is the same as the source then use a wildcard
     if [ "$expression" = "$subscription" ] || [ -z "$expression" ];
         then
         expression="."
     fi
     
     # Parsing xml depending on where the torrent url is located inside <link> tags or as the value for the attribute enclosure
     if  fgrep -iq enclosure <<< "$xmldata"
         then
         file=$(echo "$xmldata" | xsltproc parse_enclosure.xsl - 2> /dev/null)
     else
         file=$(echo "$xmldata" | xsltproc parse_link.xsl - 2> /dev/null)
     fi
     
     # Protect against the white space gotchas
     file=$(tr ' \\\t\r' '_/__' <<< "$file")
     
     for url in $file
         do
         if  echo "$url" | egrep --color -i $expression &> /dev/null
             then
             torrent=$(sed 's/\([^#]*\)#.*/\1/' <<< "$url")
             if ! fgrep -i "$torrent" catch.log > /dev/null
                 then
                 # URL and Mininova fixer
                 torrent=$(echo "$torrent" | sed -e "s/:/%3A/g" | sed -e "s/mininova.org\/tor/mininova.org\/get/g")
                 
                 echo -e "Submitting URL $torrent to $MLHOST on $(date '+(%r %D)')\n$url" >> $DEBUG
                 
                 # Submit torrent URL to mldonkey for downloading and log if sucessful 
                 wget -q -O /dev/null "http://$MLHOST/submit?q=dllink+$torrent" && echo $url >> catch.log
             fi
         fi
         # rssdata is for test matching
         echo "$url" >> rssdata
     done
done < $CONFG
bp.conf

Sample configuration file, this one will downloads from using mininova.org and revision3.com (diggnation feed). The case-insensitive regular expression string is placed before the delimiter @, if empty (or removed) it will proform a wildcard match (.*) and queue all files in the feed. It should also be possible (FileTP needs to be enable) to download regular podcasts too. 

daily.show.*(ds|TV)rip@http://www.mininova.org/rss.xml?cat=8&num=50
tw.*death.*note|eclipse.*kanon@http://www.animesuki.com/rss.php
http://revision3.com/diggnation/feed/large.xvid.avi.torrent.xml

General RSS feeds

See the sites respective FAQ and policies for information and use on their RSS feeds. See also List of BitTorrent RSS feeds (Wikipedia)

parse_enclosure.xsl

The XSL parse for RSS feeds which use the <enclosure url="..."> tag. 

<?xml version="1.0"?>
<stylesheet version="1.0"
	xmlns="http://www.w3.org/1999/XSL/Transform">
	<output method="text"/>
	<template match="/">
		<apply-templates select="/rss/channel/item"/>
	</template>
	<template match="item">
		<apply-templates select="enclosure"/>
		<text>#</text>
		<value-of select="title"/>
		<text>&#10;</text>
	</template>
	<template match="enclosure">
		<value-of select="@url"/>
	</template>
</stylesheet>
parse_link.xsl

The XSL parse for RSS feeds which use the <link>...</link> 

<?xml version="1.0"?>
<stylesheet version="1.0"
	xmlns="http://www.w3.org/1999/XSL/Transform">
	<output method="text"/>
	<template match="/">
		<apply-templates select="/rss/channel/item"/>
	</template>
	<template match="item">
		<value-of select="link"/>
		<text>#</text>
		<value-of select="title"/>
		<text>&#10;</text>
	</template>
</stylesheet>

[edit] Crontab Installation

You will need to install into your crontab file for automation. Bellow is a sample corntab that runs at different time on weekday for morning viewing (bias). 

# m hour           dom mon dow  command
00  4,7,10,14,18    *   *  1-5  /home/user/bc/broadcatch.sh
30  */3             *   *  0,6 /home/user/bc/broadcatch.sh

[edit] To do list

  • Rewrite the configuration system
    • Commenting
    • A NOT regex option
    • Episode Counter (so it doesn't re-download from different groups)
  • Options to test configuration
  • Allow regex-less config
  • Use # as a separator instead of "?" (Nicer log files)

[edit] See also


Languages:  English

Retrieved from "http://mldonkey.sourceforge.net/Broadcatch"
Personal tools