Interview with bavard about swarming/29 bug

[mbh]

I had several questions about the swarming system [and the 29 bug].
I have posted this q/a session here. If people have any more questions, perhaps they could be mentioned as replies, so other e-mails can be sent out.

From: Fabrice <e-mail withheld>

Here are some replies to what you asked. you can of course diffuse them as much as you need.
First, we are discussing about src/daemon/common/commonSwarming2.ml, the second version of the swarming system.

> 1) What is the difference between a block and a chunk?

In my terminology, which is not always consistent, a "chunk" is an segment of data for a particular network. Chunks for edonkey are 9.5 MB segments. They all have the same size. A "block" is a segment of data internally in the swarming system. Blocks don't have the same size in one file. Suppose you are downloading a file from Edonkey and Bittorrent, where the chunks in BT for that file are 1 MB-long. Then, the file will be cut internally, so that any block in the file is always completely included in a chunk. For that, you must cut the
file every 9.5 MB and every 1 MB, so that you will have:

[1MB] [1MB] [1MB] [1MB] [1MB] [1MB] [1MB] [1MB] [1MB] [0.5MB][0.5MB]
[1MB] [1MB] etc...

the 10th and 11th blocks have 0.5 MB, because the first 10 chunks are included in the 9.5 MB Edonkey chunk, and the 10th and 11th blocks must also be included in a 1 MB BT chunk.

> 2) What is the difference between the swarmer type and the type t of
> the module system? Analogous to this question, what is the difference
> between a network block and a swarmer block?

The "swarmer" type is the record corresponding to a particular file downloaded on disk. The "t" type is an interface to that record, for a particular network. So, for one file, you will have one "swarmer" record, and several "t" records, one for each network where it is downloaded. One of them is the "primary", i.e. the network whose checksums are used to know if a particular block has been correctly downloaded or not. When all the blocks have been verified with the primary checksums, the file download is finished.

"t" contains also conversion arrays between network chunk numbers and internal swarmer blocks: t.t_t2s_blocks.(i) contains the list of internal block numbers corresponding to the chunk "i" in the network, whereas t.t_s2t_blocks.(i) contains the network block number
containing the internal block "i".

> 3) What is the purpose of the uploader type?

The "uploader" is a source for a file, with which you are connected and from which you can start downloading chunks. It contains all the data attached with that source needed by the swarming system, such as "up_chunks", the chunks that are available from that source,
"up_block" which is the current chunk been downloaded, "up_complete_blocks" the list of complete chunks that can be downloaded, and "up_ranges", the ranges that have been already requested from that client and not received yet.

> Functional questions
> 1) The swarming system seems to only be truly compatible with
> protocols in which files are downloaded in discrete ranges. However,
> many protocols don't implement such ranged downloads: for instance, in
> HTTP, many files do not provide a Content-Length field in the header,
> and in FTP, you can only specify a starting point for a file, not an
> ending point.
> Can the swarming system be used to download such files efficiently?

Currently, such networks declare small chunks (I think it's about 1 MB), the problem
is indeed that they must request a range within one chunk, while they could potentially download several chunks in one request. But I think one solution would be to allow an uploader to be requested several blocks in one request (putting a list in the "up_block" field), and then add a function that would return the longest set of blocks that can be
downloaded after the block returned by "find_block".

> 2) Is it possible to write files to disk without making use of the
> swarming system?

Yes, but there is no function to extract the file from the "t" record. You could add a :
"let get_file t = t.t_file" in the module, and "val get_file : t ->
file" in the interface, so that you can use the returned file using the CommonFile functions.

> 3) There is a mention in the 2.5.29 Changelog about a serious bug.
> Several people (including myself) believe the bug is related to some
> chunks becoming corrupt while upgrading from .28 to .29. Do you have
> any more information about this serious bug, perhaps where in the code
> it occurs, or ways to reproduce it?

If I remember correctly, I think it creates a new file with a different name, and then writes to that file instead of the former one. The result is that you have two files, one with the data downloaded before amd the second with the data downloaded after. When the download is finished, it saves only one file, which contains only the data downloaded after, while the chunks which were in the former file are lost (they are not checked again, since they were supposed to be already downloaded correctly). When I noticed that behavior, I decided that the version had not been checked enough before the release, so I wanted to test it more before doing anothe release...

I hope this will help you. I can enter in more details if you need. In particular, I could
write "comments" for functions or fields on demand.

Best regards, - Fabrice

[Amorphous]

So if i understand that correctly: "the bug" is that the naming-sheme in temp got changed and that on upgrade a new file with the new sheme is created and used (instead of moving the old file) and the already verified chunks get left behind in the old file. So when the file is finished the left behind chunks are still in the old temp file and so the finished download is corrupt. Can anyone reproduce that?

[mbh]

The best way to test this would be to add a very large file (~500 MB), and perform an upgrade when this file is almost done (say 90%). If, after the file is completed, it goes back down to approx 10%, then this is the cause of the bug. A fix should be trivial.

/me adds a file to the queue

[spiralvoice]

Hi,

!!! THIS IS A PROPOSAL FOR UPDATING TO 2.5.29 CORES, USE IT AT YOUR OWN RISK !!!
!!! IF YOU LOOSE DATA YOU HAVE BEEN WARNED !!!

I finally made some tests. I took a file from my running 2.5.28r temp directory which was 88.7% completed.

This file went into the temp of vanilla CVS 2.5.29, I started the core and it created another file
with "urn:ed2k:" prepended to the filename (=MD4 hash value) which was empty(!).
So original file was named like this: ABABABABA1717171717171, new file urn:ed2k:ABABABABA1717171717171
Logfile showed this:

[Surround]

@spiralvoice:

Hehe, are you looking for someone to try that, to be sure you won't loose your files?

EDIT.: oops, you're fast ... your second post wasn't there when I wrote this ...

[spiralvoice]

[Surround]

[spiralvoice]

[Surround]

[conradj]

My temp directory typically take 55% - 60% of my disk, and my machine is full up for disks so i'm unable to go from 2.5.28 to 2.5.29 by starting up a new core and having it download from the old core.

But hey, that's just me!

--Good catch btw Surround, with the problem of mldonkey mis-reporting available chunks. That's probably why no one's asked me for friend slots since i went to 2.5.28.

cj

[Amorphous]

from the code in donkeyComplexOptions.ml (at let value_to_file ... = ... let file_diskname) i can't see why it fails. i have some lprintf's inserted in the code, but until i can test it, some time will go by.... (running downloads and exam...)

[White_FrosT]

[Amorphous]

i think i fixed the bug in revision 204 of my branch in the svn.
see https://opensvn.csie.org/viewcvs.cgi?root=mlnet&rev=204&view=rev for more infos