TorrentFlux.com Welcome, Guest. Please login or register.
03-19-2010, 18:11:43
Home Help Search Login Register donate
TorrentFlux Home | TorrentFlux Hosting

+  TorrentFlux Forums
|-+  Code Hacks and Tools
| |-+  TorrentFlux 2.1 Hacks
| | |-+  [Hack] Add wget Support V0.1
« previous next »
Pages: [1] 2  All Go Down Print
Author Topic: [Hack] Add wget Support V0.1  (Read 9092 times)
DD32
Newbie
*
Offline Offline

Posts: 16



View Profile
« on: 04-26-2006, 02:25:48 »

Shortcomings:
1. Torrent Info thing still pops up at the end of the row, clicking it will get you no-where fast, other than a page of errors.
2. No resume support yet.. I want to know if the thing is stable before i attempt to get resume working and whatnot.
3. Occasionally the file will finish downloading, but will be stuck at 98%(or something close to that) in teh interface.. <- Should be fixed.
4. NO queue support, files are downloaded as soon as the files are submitted. NO pause NO Stop(other than killing it yourself via cli)

First a note on this.. i'm not completely sure why, and dont know if its my box, or torrentflux, or something else, maybe somethings blocking read to the stat file.. i really dont know, So be warned, This is a Beta version, it seems to work, but i'd like for other peoples comments on it.. and perhaps code fixes..

Changes that need to be made:
in index.php:
Code:
Search for this:
$url_upload = getRequestVar('url_upload');

After it, but BEFORE if(! $url_upload == '') insert this:

// <DD32>: Check to see its a wget command first.
if( (! $url_upload == '') && (getRequestVar('wget') == 'true') ){
exec("nohup "._BIN_PHP." -f wget.php $url_upload ".$cfg['user']." > /dev/null &");
sleep(2); //sleep so that hopefully the other script has time to write out the stat files.
//Prevent url_upload firing:
$url_upload = '';
}
// </DD32>

Now, to add it to the GUI of the index,
I Chose to add another button along side the URL torrent upload.. If someone thinks theres a better way then this(i didnt want to add another field..) then say so and i'll consider for the next one, hopefully with better performance.

index.php
Code:
search for: <input type="Submit" value="<?php echo _GETFILE ?>">
place this after it:
<input type="hidden" name="wget" value="false">
<input type="submit"  value="WGet It" onSelect="document.getElementById('wget').value = 'true';" onClick="document.getElementById('wget').value = 'true';" onKeyDown="document.getElementById('wget').value = 'true';">
it should now look like this:
Code:
        <td>
        <hr>
        <?php echo _URLFILE ?>:<br>
        <input type="text" name="url_upload" size="50">
<input type="Submit" value="<?php echo _GETFILE ?>">
<input type="hidden" name="wget" value="false">
<input type="submit"  value="WGet It" onSelect="document.getElementById('wget').value = 'true';" onClick="document.getElementById('wget').value = 'true';" onKeyDown="document.getElementById('wget').value = 'true';">
        </td>
        </form>


next:
in config.php
Code:
add this somewhere, At the end will do:
// <DD32>
define('_BIN_WGET','/usr/bin/wget');
define('_BIN_PHP','/usr/bin/php');
// </DD32>
(Yes following b4rt's convention of external references, correct the paths there if they are wrong. it needs to be the PHP-CLI version.)

Still config.php:
Code:
Find this like:
$cfg["file_types_array"] = array("torrent");
replace with:
$cfg["file_types_array"] = array("torrent","url");
(.url files are what i have chosen to hold the http address of the fiel to be downloaded.)

next, Attached to this is 'wget.php', save it into the same fodler as index.php.
(ie. mine is stored in '/www/torrentflux/wget.php')

Hopefully now wget will work when a url is put in the torrent URL upload, and wget is clicked.
Tested with the vanilla plain version, and b4rts codebase, so it should work with both.

Can someone with a better spec box than mine test this out and tell me if its extremely slow?
I've seen apache and mysql processes spike upto 40% cpu each while using these 2, both in the vanilla version and b4rts mods, i'm not completely sure why, and not even sure if its related to this mod or not.. any pointers there would be apreciated as well.

D
(My Box: 500mhz  + 450MB ram/ Slackware-current+apache2+php5.1.1+mysql5.0.19)
I Dont see why this wouldnt work on Win32 if TF2.1 was ported over, but i can only say it'll work on linux right now Smiley
« Last Edit: 04-26-2006, 05:38:09 by DD32 » Logged
b4rt
Hero Member
*****
Offline Offline

TF Base: Linux 2.x
Posts: 543



View Profile WWW
« Reply #1 on: 04-26-2006, 03:52:30 »


to be sure cause i only flew over the code and maybe missed the info...
this hack fetches a torrent from a url and "feeds" it to flux. (like the t-flux-internal-function but only with wget)

(Yes following b4rt's convention of external references
hehe

Can someone with a better spec box than mine test this out and tell me if its extremely slow?
I've seen apache and mysql processes spike upto 40% cpu each while using these 2, both in the vanilla version and b4rts mods, i'm not completely sure why, and not even sure if its related to this mod or not.. any pointers there would be apreciated as well.
(My Box: 500mhz  + 450MB ram/ Slackware-current+apache2+php5.1.1+mysql5.0.19)

my box : dual-P2-400, 1gig ram, kernel 2.4.32, apache 2.0.55, php 4.4, mysql 4, my "distro"... once it was a debian woody .. but at that time woody was still "unstable" Wink

Logged

torrentflux-b4rt svn-trunk
Linux 2.4.34.1, Apache 2.0.59, PHP 4.4.5, Perl 5.8.8, Python 2.5, MySQL 4.1.22
DD32
Newbie
*
Offline Offline

Posts: 16



View Profile
« Reply #2 on: 04-26-2006, 04:13:28 »

to be sure cause i only flew over the code and maybe missed the info...
this hack fetches a torrent from a url and "feeds" it to flux. (like the t-flux-internal-function but only with wget)

No.

This hack allows the download of ANY file from ftp/http sites, there was a similar mod in the 2.0 series but hasnt been updated to 2.1, and seemed overkill to me.

Basically what happens is this:
1. A Ftp/Http url is placed in the "URL Torrent Load" field and the "wget it" button is clicked.
2. The modification catches the wget download before the TF-internal download function gets to it, it then passes it off to wget.php(Which is basically forked away from the script)
3. The hack sleps 2 seconds(To wait for wget.php to get started)
4. The hack then prevents the TF-internal url downloader from running IF wget was clicked.

wget.php once forked will:
1. open a wget process and
2. download whatever file it was into the users home dir(same folder that torrents go into)
3. Stay running in the background with the output of wget running into it
4. parse the output of wget
5. writing the output of wget converted into a .stat file, which TF then picks up and displays like its a torrent
6. Eventually once the download is finished the wget.php script will close off, and its jobs done.

my box : dual-P2-400, 1gig ram, kernel 2.4.32, apache 2.0.55, php 4.4, mysql 4, my "distro"... once it was a debian woody .. but at that time woody was still "unstable"
mines an AMD 500mhz.. which back then amds mhz seemed slower/less performance than intel.. but the thing does its job nicely Smiley kernel: 2.4.33 unfortunatly sitting on a 0.5Mbit line(Such is australia)
D
Logged
b4rt
Hero Member
*****
Offline Offline

TF Base: Linux 2.x
Posts: 543



View Profile WWW
« Reply #3 on: 04-26-2006, 04:55:34 »


This hack allows the download of ANY file from ftp/http sites, there was a similar mod in the 2.0 series but hasnt been updated to 2.1, and seemed overkill to me.

Basically what happens is this:
1. A Ftp/Http url is placed in the "URL Torrent Load" field and the "wget it" button is clicked.
2. The modification catches the wget download before the TF-internal download function gets to it, it then passes it off to wget.php(Which is basically forked away from the script)
3. The hack sleps 2 seconds(To wait for wget.php to get started)
4. The hack then prevents the TF-internal url downloader from running IF wget was clicked.

wget.php once forked will:
1. open a wget process and
2. download whatever file it was into the users home dir(same folder that torrents go into)
3. Stay running in the background with the output of wget running into it
4. parse the output of wget
5. writing the output of wget converted into a .stat file, which TF then picks up and displays like its a torrent
6. Eventually once the download is finished the wget.php script will close off, and its jobs done.


ah ok, now i got it. sounds nice i gonna give it a try when i have time soon (just busy hackin something together Wink.

Logged

torrentflux-b4rt svn-trunk
Linux 2.4.34.1, Apache 2.0.59, PHP 4.4.5, Perl 5.8.8, Python 2.5, MySQL 4.1.22
mb.Tab
Hero Member
*****
Offline Offline

TF Base: Linux 2.x
Posts: 574


Gotta love my Avatar !!





View Profile WWW
« Reply #4 on: 04-26-2006, 09:55:57 »

Hey thanks for your work, will give it a try asap Smiley

mb.Tab

/E

Get the following error (it was only a test file )

Code:
Error Getting the File (install_amd64_minimal_2006.0.iso.CONTENTS.torrent), Could be a Dead URL.
And it doesn´t matter if it´s ftp or http Wink...
« Last Edit: 04-26-2006, 10:12:23 by mb.Tab » Logged

Paczesiowa
Full Member
***
Offline Offline

TF Base: Linux 2.x
Posts: 177



View Profile
« Reply #5 on: 04-26-2006, 11:51:56 »

do u have allow_url_fopen ON in php.ini?
Logged

mb.Tab
Hero Member
*****
Offline Offline

TF Base: Linux 2.x
Posts: 574


Gotta love my Avatar !!





View Profile WWW
« Reply #6 on: 04-26-2006, 12:02:10 »

...damn should have mentioned that.....yes its enabled, and there are no error msgs in the apache log ...*?*...

mb.Tab
Logged

lysip
Jr. Member
**
Offline Offline

Posts: 53




View Profile
« Reply #7 on: 04-26-2006, 19:40:13 »

b4arts code base is what now?
Logged

CentOS 4.2 (kernel: 2.6.9-22.ELsmp)
TorrentFlux version - 2.1
Python version - 2.4.1
PHP Version - 4.3.9
Webserver software and version - Apache 2.0
Add-ons/Hacks - rememberme, multiup
DD32
Newbie
*
Offline Offline

Posts: 16



View Profile
« Reply #8 on: 04-26-2006, 20:39:23 »

b4arts code base is what now?
http://www.torrentflux.com/forum/index.php/topic,1265.0.html
meaning i just used his install pack to save having ot add on all the extras.

Get the following error (it was only a test file )

Code:
Error Getting the File (install_amd64_minimal_2006.0.iso.CONTENTS.torrent), Could be a Dead URL.
And it doesn´t matter if it´s ftp or http Wink...

Hmm... Recheck that you did this part correctly:
Code:
// <DD32>: Check to see its a wget command first.
if( (! $url_upload == '') && (getRequestVar('wget') == 'true') ){
exec("nohup "._BIN_PHP." -f wget.php $url_upload ".$cfg['user']." > /dev/null &");
sleep(2); //sleep so that hopefully the other script has time to write out the stat files.
//Prevent url_upload firing:
$url_upload = '';
}
// </DD32>
That last line ($url_upload = ''Wink Should mean that the following if statement thats after that command will fail to run.. I'll have a more detailed look at that later.

And yep, it'll support both FTP/HTTP downloads, ftp usernames will work like this : ftp://user:pass@host/dir/file

D
Logged
wobbles
Newbie
*
Offline Offline

Posts: 37



View Profile
« Reply #9 on: 04-26-2006, 22:39:58 »

this mod is a good addition to any tf server, it allows u to download torrents and normal files and c there stats. Will add this my tf server and let you know how it goes.


One  question though, is it possible to redirect your incoming wget downloads through a specific port? (Need to QOS this port)


Any help is welcome.
Logged
DD32
Newbie
*
Offline Offline

Posts: 16



View Profile
« Reply #10 on: 04-26-2006, 22:53:14 »

One  question though, is it possible to redirect your incoming wget downloads through a specific port? (Need to QOS this port)

I dont know if wget supports that...

Its possible to edit wget.php so that its speed is limited
I cant get to a linux box currently, so i cant provide exact steps(i'll look it up later and edit this post).

But in wget.php theres a line
Code:
$command = ......_BIN_WGET." -something......
after the ." part you can add ANY wget command line option (i'll put all this into a config directive later for V0.2 maybe) there, if you open a linux cli and type "wget --help" you'll get a list of commands.
so if theres something listed of "--Option=Value" then i'd change that line up there to this:
Code:
$command = ......_BIN_WGET." --Option=Value -something......

D
Logged
wobbles
Newbie
*
Offline Offline

Posts: 37



View Profile
« Reply #11 on: 04-26-2006, 23:23:41 »

I also used this mod and had a error when trying to wget file http://webget.crashu.info/files/webget_2.0.14.tar.gz

what could be wrong?  :'(  (had the same output as mb.tab)
Logged
DD32
Newbie
*
Offline Offline

Posts: 16



View Profile
« Reply #12 on: 04-26-2006, 23:31:32 »

I also used this mod and had a error when trying to wget file http://webget.crashu.info/files/webget_2.0.14.tar.gz

what could be wrong?  :'(  (had the same output as mb.tab)

1. Javascript must be enabled.
2. Do you hit enter or click the button?
3. If you cant think of any reason, could you send me your index.php file? theonly_HANDLE@yahoo.com.au

-=DD32=-
PS. my HANDLE is DD32
Logged
b4rt
Hero Member
*****
Offline Offline

TF Base: Linux 2.x
Posts: 543



View Profile WWW
« Reply #13 on: 04-26-2006, 23:42:14 »

just testing this... looks nice.

about load : doesnt really touch my system (leeching a kernel from kernel.org via http):
Code:
www-data  1295  1.1  0.6 13512 6368 ?        SN   06:18   0:01 /usr/bin/php -f wget.php http://www.kernel.org/pub/linux/kernel/v2.4/linux-2.4.32.tar.gz
same to mysql... both kinda idle while processing this.

what i noticed: you want that files in that location ?
Code:
drwx------    6 www-data www-data     4096 Apr 26 04:46 .BitTornado
drwxr-xr-x    2 www-data www-data     4096 Apr 27 06:20 .mrtg
drwxr-xr-x    3 www-data www-data     4096 Apr 27 06:22 .torrents
-rw-r--r--    1 www-data www-data        8 Apr 27 06:15 .torrents.url
-rw-r--r--    1 www-data www-data       66 Apr 27 06:16 .torrents_url.stat

i just dowloaded a file via ftp (ftp://ftp.kernel.org/pub/linux/kernel/v2.4/linux-2.4.32.tar.gz)
worked flawless.

awesome is it works well with my poller/mrtg Wink


just did some quick-tests and didn see any issues so far except the location of the meta-files and maybe that looked strange :
Code:
ps aux | grep -c wget
7

edit :
just saw this in my apache-error-log:
Code:
sh: cd: HOME not set
sh: cd: HOME not set
sh: cd: /usr/local/torrent/31733/: No such file or directory
/bin/cat: /usr/local/torrent/.torrents/linux_2_4_32_tar_gz_url.stat.pid: No such file or directory
/bin/cat: /usr/local/torrent/.torrents/linux_2_4_32_tar_gz_url.stat.pid: No such file or directory
/bin/cat: /usr/local/torrent/.torrents/linux_2_4_32_tar_gz_url.stat.pid: No such file or directory

regards and thanks for this one,
b4rt
« Last Edit: 04-26-2006, 23:58:20 by b4rt » Logged

torrentflux-b4rt svn-trunk
Linux 2.4.34.1, Apache 2.0.59, PHP 4.4.5, Perl 5.8.8, Python 2.5, MySQL 4.1.22
DD32
Newbie
*
Offline Offline

Posts: 16



View Profile
« Reply #14 on: 04-26-2006, 23:59:16 »

just testing this... looks nice.

what i noticed: you want that files in that location ?
Code:
drwx------    6 www-data www-data     4096 Apr 26 04:46 .BitTornado
drwxr-xr-x    2 www-data www-data     4096 Apr 27 06:20 .mrtg
drwxr-xr-x    3 www-data www-data     4096 Apr 27 06:22 .torrents
-rw-r--r--    1 www-data www-data        8 Apr 27 06:15 .torrents.url
-rw-r--r--    1 www-data www-data       66 Apr 27 06:16 .torrents_url.stat

Hmm.... they shouldnt be there at all..

Maybe this might work abit better..
Code:
instead of this:
<input type="submit"  value="WGet It" onSelect="document.getElementById('wget').value = 'true';" onClick="document.getElementById('wget').value = 'true';" onKeyDown="document.getElementById('wget').value = 'true';">

this:

<input type="button"  value="WGet It" onSelect="document.getElementById('wget').value = 'true';" onClick="document.getElementById('wget').value = 'true';this.form.sunmit();" onKeyDown="document.getElementById('wget').value = 'true';this.form.sunmit();">

Maybe those weird files were because of it firing when it shouldnt've.
(Is there a linux_2...tar_gz.url and linux_2...tar_gz_url.stat file in ./downloads/.torrents/ ?)

The load issue wasnt really what the scripts did(the php will spike up as it starts, but settle in quickly) but rather the loading of the TF index, it allways seemed to hang for awhile for me. I'm thinking this is a issue with my apache2 config/mysql config now though, mysql seems extremely slow, even when accessed via phpmyadmin on another machine.

-=DD32=-

EDIT:

edit :
just saw this in my apache-error-log:
Code:
sh: cd: HOME not set
sh: cd: HOME not set
sh: cd: /usr/local/torrent/31733/: No such file or directory
/bin/cat: /usr/local/torrent/.torrents/linux_2_4_32_tar_gz_url.stat.pid: No such file or directory
/bin/cat: /usr/local/torrent/.torrents/linux_2_4_32_tar_gz_url.stat.pid: No such file or directory
/bin/cat: /usr/local/torrent/.torrents/linux_2_4_32_tar_gz_url.stat.pid: No such file or directory

those .pid errors is the ?netcat?(Sorry forget the name atm) script.. i'll see what i can do about getting the process id to show up, and if i can i'll write it into the file.. should fix up that error.
as for the HOME not set.. i'll have a look, i  half-copied the TF torrent execution command, something i shouldntve done as it caused more issues i think.
Thats on my toto list again Smiley
« Last Edit: 04-27-2006, 00:06:36 by DD32 » Logged
Pages: [1] 2  All Go Up Print 
« previous next »
Jump to:  


Login with username, password and session length

Powered by MySQL Powered by SMF 1.1.1 | SMF © 2006, Simple Machines LLC Powered by PHP