MLdonkey gets confused by eMules rotational block requests

[spiralvoice]

(Moved this interesting thread to the dev forum)
(split from the thread "about current upload full chunks patch")

I will have a look at the patches next week.

[TripleM]

[spiralvoice]

[pango]

DonkeyClient.new_chunk checks that received requested ranges aren't already in the list of known requests (up_chunks) before adding them, so either:

• some ranges are received again after having being served and removed from up_chunks already
  
• overlapping ranges are received (requesting several times the same data, but not using exactly the same beginning and ending offsets), so above filtering fails (less likely)
  

Either way, just testing whether received requested ranges are already in the list of not yet served requests isn't enough; How do eMule peers solve the problem ? Do they enforce requests offsets to be monotonic (new accepted requests can only have offsets larger than previous ones) ? Are already served requests "blacklisted" for a while ?
Or is this an artefact of our larger output buffering ? (I doubt it; For streaming to work, sender has to go on with next request(s) before waiting for current request to be acknowleged; So it must be a common problem)

Untested patch:
ftp://ftp.berlios.de/pub/mldonkey/pango/avoid_duplicate_sending.patch

[spiralvoice]

[spiralvoice]

http://www.mldonkey.org/forum/viewtopic.php?p=42465#42465

Translation: MLdonkey 2.7.0 and Mulus 0.18.1 are also affected
by this problem. As Mulus is based in MLDonkey 2.5.16 I guess
this bug is very old and caused by a change in eMule behaviour
which MLDonkey never adapted to.
_________________
Link overview and precompiled cores here: http://mldonkey.sourceforge.net/DownloadLinks

[pango]

Mmmh then, unless there's a bug in my patch, they must do very nasty things, like sending overlapping requests with different offsets ?
If that's the case, it may be harder to fix "cleanly"...

We really need an actual communication trace to understand what's going on

[pango]

Mmmh one possible reason why my patch doesn't work: when all requests have been served (which is possible since we have large output buffers), all upload information is dropped by DonkeyFiles.send_client_block (c.client_upload <- None), so black listed requests are also forgotten.
The question is, what will break if we remove that instruction...

ftp://ftp.berlios.de/pub/mldonkey/pango/avoid_duplicate_sending_v3.patch

(Thinking of it, it could be the single reason why zones are being sent several times; And up_done_chunks logic could be superfluous).

[spiralvoice]

A block means 10240 bytes.
A zone means 184320 bytes -> 18 blocks
A chunk means 9728000 bytes -> 52,78 zones

Here is a log from MLDonkey -> eMule 0.47c data transfer:

Explanation of the log:

[pango]

To test my hypothesis that client_upload <- None is the real culpit, I made a different patch with all the up_done_chunks logic removed:

ftp://ftp.berlios.de/pub/mldonkey/pango/avoid_duplicate_sending_v4.patch

[spiralvoice]

[pango]

[pango]

[spiralvoice]

[pango]

500000 is not large enough to store 3 zones (540kB), but on a fast-enough link, the third can be send (and the requested zones list emptied) before the next batch of requests arrives. On my 128kbps uplink, I never get the "NO CHUNKS" messages in the logs, and streaming works fine.