persistent torrent-totals hack

[b4rt]

a hack to have persistent totals-stats per torrent over client-restart-cycles.
(totals are saved to database for every torrent.)
this again is a piece of quickhack-code even lacking error-handling. ~

the hack is only tested on tf-2.1-b4rt-6 but it should work on any tf 2.1
installation although i am not soo sure where to integrate the snips exactly
because my version is already very different in that parts. (my current version
is already using the client-handler-architecture and has part of this hack
integrated there...)
means the places i describe here are actually never tested, but as this hack is
rather trivial integration should be quite easy.

1. database
choose the snip for your database and import into your torrentflux database.

1.1 mysql

Code:

--
-- tf_torrent_totals
--
CREATE TABLE tf_torrent_totals (
  tid VARCHAR(40) NOT NULL default '',
  uptotal BIGINT(80) NOT NULL default '0',
  downtotal BIGINT(80) NOT NULL default '0',
  PRIMARY KEY  (tid)
) TYPE=MyISAM;

1.2 non-mysql
(as always this version is untested ~)

Code:

--
-- tf_torrent_totals
--
CREATE TABLE tf_torrent_totals (
  tid VARCHAR(40) NOT NULL default '',
  uptotal INTEGER(80) NOT NULL default '0',
  downtotal INTEGER(80) NOT NULL default '0',
  PRIMARY KEY  (tid)
) ;

2. new functions
add where you have your global functions defined.. (eg functions.php)

Code:

/**
 * gets totals of a torrent
 *
 * @param $torrent name of the torrent
 * @return array with torrent-totals
 */
function getTorrentTotals($torrent) {
    global $cfg, $db;
    if ( !isset($torrent) || !preg_match('/^[a-zA-Z0-9._]+$/', $torrent) )
        return;
$torrentId = getTorrentHash($torrent);
    $sql = "select uptotal,downtotal from tf_torrent_totals where tid = '".$torrentId."'";
$retVal = array();
    $result = $db->Execute($sql);
$row = $result->FetchRow();
if (!empty($row)) {
$retVal["uptotal"] = $row["uptotal"];
$retVal["downtotal"] = $row["downtotal"];
} else {
$retVal["uptotal"] = 0;
$retVal["downtotal"] = 0;
}
return $retVal;
}

/**
 * updates totals of a torrent
 *
 * @param $torrent name of the torrent
 * @param $uptotal uptotalof the torrent
 * @param $downtotal downtotal of the torrent
 */
function updateTorrentTotals($torrent, $uptotal, $downtotal) {
    global $cfg, $db;
    if ( !isset($torrent) || !preg_match('/^[a-zA-Z0-9._]+$/', $torrent) )
        return;
$torrentId = getTorrentHash($torrent);
    $sql = "select uptotal,downtotal from tf_torrent_totals where tid = '".$torrentId."'";
    $result = $db->Execute($sql);
    $row = $result->FetchRow();
    if (!empty($row)) {
        $currentUp           = $row["uptotal"];
        $currentDown         = $row["downtotal"];
$upSum = $currentUp + $uptotal;
$downSum = $currentDown + $downtotal;
$sql = "UPDATE tf_torrent_totals SET uptotal = '".($upSum+0)."', downtotal = '".($downSum+0)."' WHERE tid = '".$torrentId."'";
$db->Execute($sql);
    } else {
$sql = "INSERT INTO `tf_torrent_totals` ( `tid` , `uptotal` ,`downtotal` )
            VALUES (
                    '".$torrentId."',
                    '".$uptotal."',
                    '".$downtotal."'
                   )";
$db->Execute($sql);
}
}

/**
 * gets hash of a torrent
 *
 * @param $torrent name of the torrent
 * @return var with torrent-hash
 */
function getTorrentHash($torrent) {
global $cfg;
$result = shell_exec("cd " . $cfg["torrent_file_path"]."; " . $cfg["pythonCmd"] . " -OO " . $cfg["btshowmetainfo"]." \"".$torrent."\"");
$resultAry = explode("\n",$result);
$hashAry = explode(":",trim($resultAry[3]));
return trim($hashAry[1]);
}

3. integration

3.1 downloaddetails.php

find :

Code:

{
    die("fatal error torrent file not specified");
}

add below :

Code:

$divi = 1024<<10;
$totalAry = getTorrentTotals(urldecode($torrent));
$upTotalCurrent = $af->uptotal/$divi;
$downTotalCurrent = $af->downtotal/$divi;
$upTotal = $totalAry["uptotal"]/$divi;
$downTotal = $totalAry["downtotal"]/$divi;
$upTotal += $upTotalCurrent;
$downTotal += $downTotalCurrent;
if ($af->running != 1) {
    $upTotalCurrent = 0;
    $downTotalCurrent = 0;
}

find :

Code:

        <tr>
            <td align="right"><div class="tiny">Down:</div></td>
            <td bgcolor="<?php echo $cfg["body_data_bg"] ?>"><div class="tiny"><?php echo "<strong>".formatFreeSpace($af->GetRealDownloadTotal())."</strong>" ?></div></td>
            <td align="right"><div class="tiny">Up:</div></td>
            <td bgcolor="<?php echo $cfg["body_data_bg"] ?>"><div class="tiny"><?php echo "<strong>".formatFreeSpace($af->uptotal/(1024*1024))."</strong>" ?></div></td>
        </tr>

replace with :

Code:

        <tr>
            <td align="right"><div class="tiny">Down:</div></td>
            <td bgcolor="<?php echo $cfg["body_data_bg"] ?>"><div class="tiny"><?php echo "<strong>".formatFreeSpace($downTotalCurrent)."</strong>" ?></div></td>
            <td align="right"><div class="tiny">Up:</div></td>
            <td bgcolor="<?php echo $cfg["body_data_bg"] ?>"><div class="tiny"><?php echo "<strong>".formatFreeSpace($upTotalCurrent)."</strong>" ?></div></td>
        </tr>
        <tr>
            <td align="right"><div class="tiny">Down-Total:</div></td>
            <td bgcolor="<?php echo $cfg["body_data_bg"] ?>"><div class="tiny"><?php echo "<strong>".formatFreeSpace($downTotal)."</strong>" ?></div></td>
            <td align="right"><div class="tiny">Up-Total:</div></td>
            <td bgcolor="<?php echo $cfg["body_data_bg"] ?>"><div class="tiny"><?php echo "<strong>".formatFreeSpace($upTotal)."</strong>" ?></div></td>
        </tr>

3.2 where you start your torrents (thats in index.php on 2.1 final)
may work on 2.1 final :

find :

Code:

@unlink($cfg["torrent_file_path"].$delfile);

add above :

Code:

include_once("AliasFile.php");
$af = new AliasFile($cfg['torrent_file_path'].$alias_file, 0);
// update totals for this torrent
updateTorrentTotals($delfile, $af->uptotal+0, $af->downtotal+0);

3.3 where you delete your torrents (thats in index.php on 2.1 final)
may work on 2.1 final :

find :

Code:

// create AliasFile object and write out the stat file
$af = new AliasFile($cfg["torrent_file_path"].$alias.".stat", $owner);

if ($cfg["AllowQueing"])
{
    if($queue == "1")
    {
        $af->QueueTorrentFile();  // this only writes out the stat file (does not start torrent)
    }
    else
    {
        $af->StartTorrentFile();  // this only writes out the stat file (does not start torrent)
    }
}

replace with :

Code:

// create AliasFile object and write out the stat file
$af = new AliasFile($cfg["torrent_file_path"].$alias.".stat", $owner);
// update totals for this torrent
updateTorrentTotals($torrent, $af->uptotal+0, $af->downtotal+0);
if ($cfg["AllowQueing"])
{
    if($queue == "1")
    {
        $af->QueueTorrentFile();  // this only writes out the stat file (does not start torrent)
    }
    else
    {
        $af->StartTorrentFile();  // this only writes out the stat file (does not start torrent)
    }
}

i attached a screenie of the downloaddetails of a test-torrent which was stopped and resumed multiple times.


regards,
b4rt

[ryaner]

Dude, your coding some very very nice hacks. Saving the stats like this would also help with the xfer hack problems. One thing you may like to add to this is the ability to work out the seed until ratio correctly when restarting a torrent. Currently BitTornado will use the total size of a torrent when determining when to die.

e.g.
You are downloading a 100mb torrent and want to seed 200% total.
Stop and restart after uploading 50mb of data.
Leaving it at 200% causes it to upload a further 200mb of data as the download total resets upon completion so 150% should have been passed on restart.

Edit: This works straight into TF2.1 Final too.

[b4rt]

Dude, your coding some very very nice hacks.

hehe, thx. still very php-unsafe but its gettin better ~

Saving the stats like this would also help with the xfer hack problems.

i read in the forum there is a problem with xfer but didnt dig into it so far so i dont know where the problem is or how this hack could help. this hack is far less complex than the xfer-hack.

One thing you may like to add to this is the ability to work out the seed until ratio correctly when restarting a torrent. Currently BitTornado will use the total size of a torrent when determining when to die.
e.g.
You are downloading a 100mb torrent and want to seed 200% total.
Stop and restart after uploading 50mb of data.
Leaving it at 200% causes it to upload a further 200mb of data as the download total resets upon completion so 150% should have been passed on restart.

didnt think of anything like that so far, primary goal was to safe my stats. (so annoying to loose seed-totals on client restart ~) (and i use only "seed forever" so i didnt think of this out of personal need so far)
if i understand right your idea is to "re-calc" the percentage-val for "sharekill" on client-start.
this done with the help from the persistent torrent-totals-values + data-size.
one hurdle to take is that the initial (wanted) sharekill-val (the 200% in your example) is overwritten on client-restarts with current percentage-value. (so storing the "wanted value" in another place might be neccessary)
ill look into this tomorrow when i have time, i am too sleepy already

regards,
b4rt

[ryaner]

Saving the stats like this would also help with the xfer hack problems.

i read in the forum there is a problem with xfer but didnt dig into it so far so i dont know where the problem is or how this hack could help. this hack is far less complex than the xfer-hack.

The xfer hack suffers from minus values all over the place. Using this method, storing minus values would become unnecessary if people would accept that some daily stats would be off if they hadn't got the index page open to update them. A cronjob a 12 every night would sort that out very quickly though.

One thing you may like to add to this is the ability to work out the seed until ratio correctly when restarting a torrent. Currently BitTornado will use the total size of a torrent when determining when to die.
e.g.
You are downloading a 100mb torrent and want to seed 200% total.
Stop and restart after uploading 50mb of data.
Leaving it at 200% causes it to upload a further 200mb of data as the download total resets upon completion so 150% should have been passed on restart.

didnt think of anything like that so far, primary goal was to safe my stats. (so annoying to loose seed-totals on client restart ~) (and i use only "seed forever" so i didnt think of this out of personal need so far)
if i understand right your idea is to "re-calc" the percentage-val for "sharekill" on client-start.
this done with the help from the persistent torrent-totals-values + data-size.
on hurdle to take is that the initial (wanted) sharekill-val (the 200% in your example) is overwritten on client-restarts with current percentage-value. (so storing the "wanted value" in another place might be neccessary)
ill look into this tomorrow when i have time, i am too sleepy already

Ye storing the input value would be necessary so that the value passed to BitTornado could be recalculated each time. Should be simple enough I think.

[b4rt]

the problem is more a presentation-problem than a implementation problem.

following scenario:

• you start a fresh torrent with interactive start. you choose 200% sharekill
• torrent runs, 50% are processed. you stop it
• you resume the torrent with interactive start (the "presentation-problem" is of course only present when resuming interactive)
  the sharekill-field gets prefilled with your original set value, 200%. you think "thats what i wanted and still want" and start the torrent. i guess you see the problem... if you resubmit form with 200% sharekill-var you "reset" your sharekill-percentage
• showing the recalced percentage-var in the form when resuming would be even more irritating (and somehow "wrong")
• i discourage multiple text-fields as workaround. irritating and ugly

any thoughts welcome,
b4rt

[ryaner]

Perhaps just a tick box beside the sharekill value for calculating over share kill value?

[b4rt]

Perhaps just a tick box beside the sharekill value for calculating over share kill value?

hmm extra-checkbox sounds not soo bad
but maybe a check-box like "force set value" would be more clear.
so the sharekill-value is only "set" when checkbox is activated.
(and it wont be activated by default when resuming a torrent)
this way there would not be the need to safe an extra-percentage-value because the "current" one is calced on start and the presented one is the one already safed in the torrent-settings.

on the other hand this would completely override current flux-behaviour because the "over share kill" would be used always.
but seeing that from a objective point of view... i would say it is not such a bad thing as imho the flux-behaviour (better : tornado,bittorrent,...) is not the behaviour of the heavy weight clients. (eg azureus). these clients only use the "over share kill" method.

[PiForever]

damn! i was going to request something like this...speechless...5/5

[b4rt]

Perhaps just a tick box beside the sharekill value for calculating over share kill value?

hmm extra-checkbox sounds not soo bad
but maybe a check-box like "force set value" would be more clear.
so the sharekill-value is only "set" when checkbox is activated.
(and it wont be activated by default when resuming a torrent)
this way there would not be the need to safe an extra-percentage-value because the "current" one is calced on start and the presented one is the one already safed in the torrent-settings.

on the other hand this would completely override current flux-behaviour because the "over share kill" would be used always.
but seeing that from a objective point of view... i would say it is not such a bad thing as imho the flux-behaviour (better : tornado,bittorrent,...) is not the behaviour of the heavy weight clients. (eg azureus). these clients only use the "over share kill" method.

i implemented this in my current version and must admit that was a nice idea ryaner. (especially that with recalcing percentage-param)

downside is its not a standalone-hack anymore because it depends on some of my additions.
the percentage-recalc-code is implemented in the clienthandler-subclass of tornado as its more a "proprietary" feature afaic. (its fine if other handlers implement it as well. param is a field of base-class.) thinkable would be to move the recalc-code into the clienthandler-base class if other clients have the same param-type for this value. but because i dont know that now it is in the tornado-handler itself atm

current implementation :

• acts like a heavy client (eg azureus) at the "sharekill" setting. -> real ratio-calculations (over client-starts).
• scenario:
  
  • you start a torrent with 200% sharekill.
  • you stop it after n %.
  • you resume it.
  • -> torrent will seed until "real upload" reaches 200%, not until the upload of the current "session" (the client) reaches 200%.
• additional it is possible to re-set the sharekill value on (interactive) torrent-restart if you decide to change it.
• sharing-percentage in downloaddetails is calculated from "real" upload, not only from current upload.
• the value seen as sharekill (seed until) in userland is the value the user has set for a torrent. ("global" and persistent. -> over client-restarts) the actual real value the client(s) are started with are not visible in userland and may have not the same value as the sharekill-value.

i attached some screenies to show how it looks atm in userland :

• flux-torrent-totals-1.jpg : start-dialog of a fresh torrent
• flux-torrent-totals-2.jpg : downloaddetails of torrent just after first start
• flux-torrent-totals-3.jpg : downloaddetails of torrent after it was running for a while
• flux-torrent-totals-4.jpg : downloaddetails of torrent after i stopped it, resumed it and again it was running for a while
• flux-torrent-totals-5.jpg : downloaddetails of torrent after i stopped it again, resumed it again and again it was running for a while

[ryaner]

Very nicely done. Must hack this into my own version now.

Edit: Would you mind uploading the code somewhere

[b4rt]

Very nicely done. Must hack this into my own version now.

Edit: Would you mind uploading the code somewhere

lol i was just asking if you want the code while you edited it ~
you want snips of the core-parts or whole files involved ?

[ryaner]

Whatever is easy for yourself. I'll be able work out whats needed to change to fit into my version.

[b4rt]

there are only 4 files (in my version ~) involved.
ill attach em and provide some snips here to make things easier.

1. startpop.php

add a checkbox for force-set sharekill value. (also initial set)

Code:

            <td colspan="4" align="center">
             <div ID="sharekiller" align="center" style="visibility:hidden;">
              Keep seeding until Sharing is: <input type="Text" name="sharekill" maxlength="4" size="4" value="<?php echo $cfg["sharekill"]; ?>">%<font class="tiny">  (0% will keep seeding)</font>
              &nbsp;
              <?php
                echo "<input type='Checkbox' name='sharekill_forceset'";
                if (! $torrentExists)
                    echo " checked";
                echo ">(re-)set sharekill";
              ?>
             </div>
            </td>

2. downloaddetails.php

just the snips from core-action. you sure know how/where to present it.

Code:

// Load saved settings
loadTorrentSettingsToConfig($torrent);
// totals
$upTotalCurrent = $af->uptotal;
$downTotalCurrent = $af->downtotal;
$totalAry = getTorrentTotals(urldecode($torrent));
$upTotal = $totalAry["uptotal"];
$downTotal = $totalAry["downtotal"];
$upTotal += $upTotalCurrent;
$downTotal += $downTotalCurrent;
// seeding-%
$torrentSize = $af->size+0;
$sharing = number_format((($upTotal / $torrentSize) * 100), 2);
$torrent_port = "";
$torrent_cons = "";
$label_max_download_rate = "";
$label_max_upload_rate = "";
$label_downTotal = formatFreeSpace($downTotal / 1048576);
$label_upTotal = formatFreeSpace($upTotal / 1048576);
$label_downTotalCurrent = "";
$label_upTotalCurrent = "";
$label_seeds = "";
$label_peers = "";
$label_maxcons = "";
$label_sharing = $sharing . '%';
$label_sharekill = $cfg["sharekill"] . '%';
if (($af->running == 1) && ($alias != "")) {
    $label_downTotalCurrent = formatFreeSpace($downTotalCurrent / 1048576);
    $label_upTotalCurrent = formatFreeSpace($upTotalCurrent / 1048576);
    $label_seeds = $af->seeds;
    $label_peers = $af->peers;
    $torrent_pid = getTorrentPid($alias);
    $torrent_port = netstatPortByPid($torrent_pid);
    $torrent_cons = netstatConnectionsByPid($torrent_pid);
    $label_max_download_rate = " (".number_format($cfg["max_download_rate"], 2).")";
    $label_max_upload_rate = " (".number_format($cfg["max_upload_rate"], 2).")";
    $label_maxcons = " (".$cfg["maxcons"].")";
}

3. where you process the request-vars from torrent-start-dialog

Code:

            $sharekill_forceset = getRequestVar('sharekill_forceset');
            if (empty($sharekill_forceset)) { // no force set sharekill
                // we need to load the share-kill-value from the stored settings
                $settingsAry = loadTorrentSettings(urldecode($torrent));
                $this->sharekill = $settingsAry["sharekill"];
                // following should never happen... but better ensure a suitable
                // fallback. maybe user unchecked checkbox on a fresh torrent
                if ((! isset($this->sharekill)) || (empty($this->sharekill)))
                    $this->sharekill = 0;
            } else { // force set sharekill
                $this->sharekill = getRequestVar('sharekill');
                if ($this->runtime == "True" )
                    $this->sharekill = "-1";
                if (empty($this->sharekill)) {
                    if ($this->sharekill != "0") {
                        $this->sharekill = $this->cfg["sharekill"];
                    }
                }
            }

4. where you build the command-line for a tornado-client-start :

Code:

        // set param for sharekill
        if ($this->sharekill <= 0) { // nice, we seed forever
            $this->sharekill_param = 0;
        } else { // recalc sharekill
            $totalAry = getTorrentTotals(urldecode($torrent));
            $upTotal = $totalAry["uptotal"]+0;
            $torrentSize = $this->af->size+0;
            $upWanted = ($this->sharekill / 100) * $torrentSize;
            if ($upTotal >= $upWanted) { // we already have seeded at least
                                         // wanted percentage. continue to seed
                                         // forever is suitable in this case ~~
                $this->sharekill_param = 0;
            } else { // not done seeding wanted percentage
                $this->sharekill_param = (int) ($this->sharekill - (($upTotal / $torrentSize) * 100));
                // the type-cast may have floored the value. (tornado lacks
                // precision because only (really?) accepting percentage-values)
                // better to seed more than less so we add a percent in case ;)
                if (($upWanted % $upTotal) != 0)
                    $this->sharekill_param += 1;
                // sanity-check.
                if ($this->sharekill_param <= -1)
                    $this->sharekill_param = 0;
            }
        }

EDIT: erm i just realized its not really all you need. you need my torrent-settings-persistance-code as well. the code and the db-table for this hasnt changed since v5 so you might look into the code there.
(or simply adapt to your own persistance code as you sure have one because the idea for mine was from yours )


good luck,
b4rt

[ryaner]

Have the bits of this I wanted hacked into my code and came across a small maths bug.

Code:

This line
$sharekill = (int) ($sharekill - (($upTotal / $upWanted) * 100));

should be
$sharekill = (int) ($sharekill - (($upTotal / $torrentSize) * 100));

Was causing the wrong percentages to be generated.

[ryaner]

One other thing, when deleting, the torrent should prob be removed from the database to clean up after itself and not just have the totals updated.