MLDonkey Forum Index
Homepage •  Bugs •  Tasks •  Patches •  SF.net Project Page •  ChangeLog •  German forum •  Links •  Wiki •  Downloads
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 
 
Stupid new user questions

 
Post new topic   Reply to topic    MLDonkey Forum Index -> Problems with MLDonkey Client (in general, and ed2k, kad, overnet)
View previous topic :: View next topic  
Author Message
stoatwblr
neophyte


Joined: 10 Jun 2013
Posts: 9

PostPosted: Mon Jun 10, 2013 12:56 am    Post subject: Stupid new user questions Reply with quote

I've run up mldonkey after many years of using amule and have a few questions that I can't see answered in the faqs or forums.

1: Is there any way of speeding up the initial hashing?
2: What's the maximum number of local files (uploads) supported?

I have something like 34k files to hash (about 14Tb), so it's taking a while. Overflowing a buffer wouldn't help things...

3 - more for interest - is there a timeframe for supporting obfuscation? A lot of emule/amule users have it enforced now, so they're uncontactable as things stand.

Thanks in advance
Back to top
View user's profile Send private message
ygrek
professional


Joined: 20 Mar 2010
Posts: 594

PostPosted: Mon Jun 10, 2013 10:26 am    Post subject: Reply with quote

1. I fear not without code changes. How much cpu is used and what is the speed of hashing?
2. There is no limit in mldonkey that I know of, but the number of open files and sockets is limited by the OS, you should probably raise rlimits if you want to sahre many files.
3. Do not count on me for this - I have no experience with edk protocol and no incentive to reverse engineer it.
_________________
Download | Report bugs | git mirror
Back to top
View user's profile Send private message Visit poster's website
stoatwblr
neophyte


Joined: 10 Jun 2013
Posts: 9

PostPosted: Mon Jun 10, 2013 6:19 pm    Post subject: Reply with quote

ygrek wrote:
1. I fear not without code changes. How much cpu is used and what is the speed of hashing?


That's the irritating thing. Only about 15% of one Xeon 5140 is being used (there are 4 processors available across 2 CPUs). Hashing is running at about 500Mb/min. (This is on UbuntuRaring/ZFS with enough memory to avoid swapping and a decent size ssd l2arc)

Quote:
2. There is no limit in mldonkey that I know of, but the number of open files and sockets is limited by the OS, you should probably raise rlimits if you want to sahre many files.


That's set to unlimited, but so far (4000 files hashed) it's only grown to about 200Mb, vs Amuled being 1Gb VM for 5,000 files.

Quote:
3. Do not count on me for this - I have no experience with edk protocol and no incentive to reverse engineer it.


There is some documentation for it. If you're into unpicking CPP then the amule code has a lot of comments in it (part of the reason I'm moving to MLD is that amule has effectively been abandoned) and may be importable. That's the beauty of Opensauce after all Smile

I can see your PoV about ed2k/kad though. It cannot run in an IPv6 network as currently implemented, so probably has a lifespan of less than 3-4 years left.


Where are the hashtables kept? Are they really the shared_file.ini files in /var/lib/mldonkey or are they held somewhere else?
Back to top
View user's profile Send private message
ygrek
professional


Joined: 20 Mar 2010
Posts: 594

PostPosted: Mon Jun 10, 2013 8:44 pm    Post subject: Reply with quote

stoatwblr wrote:

That's the irritating thing. Only about 15% of one Xeon 5140 is being used (there are 4 processors available across 2 CPUs). Hashing is running at about 500Mb/min. (This is on UbuntuRaring/ZFS with enough memory to avoid swapping and a decent size ssd l2arc)

I will try to look into improving this.
Quote:

Where are the hashtables kept? Are they really the shared_file.ini files in /var/lib/mldonkey or are they held somewhere else?

Everything is stored in ini files.
_________________
Download | Report bugs | git mirror
Back to top
View user's profile Send private message Visit poster's website
stoatwblr
neophyte


Joined: 10 Jun 2013
Posts: 9

PostPosted: Tue Jun 11, 2013 12:39 am    Post subject: Reply with quote

ygrek wrote:
stoatwblr wrote:

That's the irritating thing. Only about 15% of one Xeon 5140 is being used (there are 4 processors available across 2 CPUs). Hashing is running at about 500Mb/min.


I will try to look into improving this


2 thing which probably help would help a lot:

Periodically write shared_files_new.ini back into shared files (every N files or N minutes is a reasonable compromise).

On a clean shutdown, do the same thing before exiting.

That way it shouldn't have to start over if hashing didn't complete (Checkpointing may seem a waste of time when prototyping, but when you end up having to rerun large batch jobs, it makes more sense. Same principle applies here)

Thanks for taking an interest Smile
Back to top
View user's profile Send private message
stoatwblr
neophyte


Joined: 10 Jun 2013
Posts: 9

PostPosted: Wed Jul 03, 2013 8:49 pm    Post subject: Reply with quote

Apologies for the assumptions. it was a bit startling at first to see the shared counter drop from 8000 back to 200 after a crash. I see it's doing exactly that (the writeback/periodic writes)

However, it does seem that the filesystem scanning code is pretty slow. It's only stat()ing about 20 files per second at best. A nasty little shell script I whipped up can do the whole lot (41,000 files) and present a list of files to (re)hash in about 5 seconds so it doesn't seem that IO is the holdup

Any idea why the reshare code is so slow?
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic    MLDonkey Forum Index -> Problems with MLDonkey Client (in general, and ed2k, kad, overnet) All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Sourceforge.net Logo