MLDonkey Forum Index
Homepage •  Bugs •  Tasks •  Patches •  SF.net Project Page •  ChangeLog •  German forum •  Links •  Wiki •  Downloads
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 
 
Uncontrollable increasing CPU usage

 
Post new topic   Reply to topic    MLDonkey Forum Index -> Problems with MLDonkey Client (in general, and ed2k, kad, overnet)
View previous topic :: View next topic  
Author Message
dennisn
user


Joined: 08 Sep 2004
Posts: 81
Location: Canada

PostPosted: Thu Oct 10, 2013 1:04 am    Post subject: Uncontrollable increasing CPU usage Reply with quote

Any idea what's causing my mlnet to steadily (almost linearly) increase my CPU usage, to the point where I have to restart it about every day now?




I don't see any related similar increase in IO usage or memory usage -- just CPU. "mlnet.log" doesn't show anything obviously wrong, except things like:

Code:
 HTTPsv 81.163.16.199:50300 blocked: Unknown
 HTTPsv 81.163.16.199:50300 blocked: Unknown
 HTTPsv connection from 81.163.16.199 rejected (IP is blocked)


... every few minutes. (That address being listed/blocked from my level1.gz file.)

Here are the heads of a bunch of gprof profiling runs that I did:

Code:
   %   cumulative   self              self     total
 time   seconds   seconds    calls  Ks/call  Ks/call  name

 26.87   1085.69  1085.69  5850092     0.00     0.00  camlBasicSocket__exec_tasks_1550
  9.41   1465.93   380.24  5850091     0.00     0.00  camlBasicSocket__iter_task_1504
  7.79   1780.44   314.51 758435396     0.00     0.00  caml_modify
  5.54   2004.37   223.93 11700183     0.00     0.00  try_poll
  4.80   2198.27   193.90   622991     0.00     0.00  mark_slice
  3.84   2353.41   155.14 47379276     0.00     0.00  camlQueue__fold_1087
  2.58   2457.74   104.33  1249588     0.00     0.00  sweep_slice
  2.31   2550.88    93.14  3744929     0.00     0.00  caml_empty_minor_heap
  2.21   2640.26    89.38 2806490461     0.00     0.00  caml_fl_merge_block
  2.20   2729.02    88.76   650544     0.00     0.00  camlCommonSwarming__intervals_to_string_3017


 29.27    715.60   715.60  5189055     0.00     0.00  camlBasicSocket__exec_tasks_1550
 11.52    997.13   281.53  5189054     0.00     0.00  camlBasicSocket__iter_task_1504
  8.85   1213.40   216.27 159669569     0.00     0.00  caml_modify
  7.48   1396.29   182.89 10378109     0.00     0.00  try_poll
  4.80   1513.55   117.26 35691684     0.00     0.00  camlQueue__fold_1087
  3.68   1603.60    90.05   352967     0.00     0.00  mark_slice
  1.96   1651.63    48.03 4090789436     0.00     0.00  caml_apply2
  1.69   1692.86    41.23 2781966036     0.00     0.00  caml_darken


 15.66    192.06   192.06  3221548     0.00     0.00  camlBasicSocket__exec_tasks_1550
  7.17    279.98    87.92 26613262     0.00     0.00  camlQueue__fold_1087
  6.98    365.65    85.67 3479318823     0.00     0.00  caml_modify
  6.86    449.86    84.21  3221547     0.00     0.00  camlBasicSocket__iter_task_1504
  4.51    505.16    55.30   193640     0.00     0.00  mark_slice
  4.26    557.45    52.29  6443095     0.00     0.00  try_poll
  3.50    600.44    42.99 33367554     0.00     0.00  sha1_compile
  3.01    637.42    36.98 3142171429     0.00     0.00  caml_apply2
  2.07    662.80    25.38   379073     0.00     0.00  camlCommonSwarming__intervals_to_string_3017


 21.82    405.82   405.82  4239284     0.00     0.00  camlBasicSocket__exec_tasks_1550
  8.55    564.86   159.04  4239283     0.00     0.00  camlBasicSocket__iter_task_1504
  7.63    706.73   141.87 1458659928     0.00     0.00  caml_modify
  6.14    820.85   114.12 34330060     0.00     0.00  camlQueue__fold_1087
  5.34    920.14    99.29  8478567     0.00     0.00  try_poll
  4.25    999.18    79.04   289550     0.00     0.00  mark_slice
  2.54   1046.34    47.16 4021199647     0.00     0.00  caml_apply2
  2.31   1089.34    43.00 33379734     0.00     0.00  sha1_compile
  1.79   1122.72    33.38   582956     0.00     0.00  sweep_slice


 19.44    359.42   359.42  4621749     0.00     0.00  camlBasicSocket__exec_tasks_1550
  8.23    511.58   152.16  4621748     0.00     0.00  camlBasicSocket__iter_task_1504
  7.48    649.79   138.21 1251044006     0.00     0.00  caml_modify
  6.32    766.57   116.78 35618326     0.00     0.00  camlQueue__fold_1087
  5.23    863.32    96.75  9243497     0.00     0.00  try_poll
  4.19    940.87    77.55   293631     0.00     0.00  mark_slice
  2.76    991.99    51.12 49125272     0.00     0.00  caml_apply2
  2.67   1041.33    49.34 38834004     0.00     0.00  sha1_compile


 26.39    846.27   846.27  6372588     0.00     0.00  camlBasicSocket__exec_tasks_1550
  9.87   1162.76   316.49  6372587     0.00     0.00  camlBasicSocket__iter_task_1504
  7.66   1408.45   245.69 1204052506     0.00     0.00  caml_modify
  6.10   1603.99   195.54 12745175     0.00     0.00  try_poll
  5.07   1766.70   162.71 49022927     0.00     0.00  camlQueue__fold_1087
  3.98   1894.44   127.74   463799     0.00     0.00  mark_slice
  2.45   1973.06    78.62 62877264     0.00     0.00  sha1_compile
  2.16   2042.21    69.15 1605042607     0.00     0.00  caml_apply2


 26.01   1197.11  1197.11  5478924     0.00     0.00  camlBasicSocket__exec_tasks_1550
 14.52   1865.33   668.22  5478923     0.00     0.00  camlBasicSocket__iter_task_1504
  8.91   2275.27   409.94 2424215842     0.00     0.00  caml_modify
  8.41   2662.54   387.27 10957848     0.00     0.00  try_poll
  4.35   2862.91   200.37   633395     0.00     0.00  mark_slice
  3.17   3008.62   145.71 43788230     0.00     0.00  camlQueue__fold_1087
  2.36   3117.09   108.47  1173283     0.00     0.00  sweep_slice
  2.26   3221.08   103.99  3613147     0.00     0.00  caml_empty_minor_heap
  2.12   3318.60    97.52 2474412863     0.00     0.00  caml_fl_merge_block
  1.98   3409.92    91.32 2548985528     0.00     0.00  caml_oldify_one


 32.70    967.58   967.58  4177047     0.00     0.00  camlBasicSocket__exec_tasks_1550
 12.60   1340.31   372.73  4177046     0.00     0.00  camlBasicSocket__iter_task_1504
  8.06   1578.82   238.51 119483106     0.00     0.00  caml_modify
  7.70   1806.51   227.69  8354092     0.00     0.00  try_poll
  3.75   1917.32   110.81   360533     0.00     0.00  mark_slice
  3.46   2019.56   102.24 30892214     0.00     0.00  camlQueue__fold_1087
  2.01   2079.03    59.47   691681     0.00     0.00  sweep_slice
  1.97   2137.37    58.34 1343339616     0.00     0.00  caml_fl_merge_block
  1.88   2193.03    55.66  2104283     0.00     0.00  caml_empty_minor_heap
  1.65   2241.86    48.83 2863737170     0.00     0.00  caml_darken


 25.59    275.67   275.67  2244502     0.00     0.00  camlBasicSocket__exec_tasks_1550
 10.02    383.63   107.96  2244501     0.00     0.00  camlBasicSocket__iter_task_1504
  7.87    468.41    84.78 3311323690     0.00     0.00  caml_modify
  6.37    537.00    68.59  4489003     0.00     0.00  try_poll
  4.84    589.16    52.16 15987416     0.00     0.00  camlQueue__fold_1087
  3.40    625.84    36.68   143252     0.00     0.00  mark_slice
  3.09    659.18    33.34 26234140     0.00     0.00  sha1_compile
  2.07    681.52    22.34 1855783507     0.00     0.00  caml_apply2


 31.56    309.44   309.44  1774004     0.00     0.00  camlBasicSocket__exec_tasks_1550
 11.10    418.25   108.81  1774003     0.00     0.00  camlBasicSocket__iter_task_1504
  7.95    496.17    77.92 3140785288     0.00     0.00  caml_modify
  7.22    567.01    70.84  3548007     0.00     0.00  try_poll
  4.19    608.13    41.12 12917641     0.00     0.00  camlQueue__fold_1087
  3.04    637.89    29.76   135739     0.00     0.00  mark_slice
  1.88    656.30    18.41 1491058646     0.00     0.00  caml_apply2
  1.55    671.45    15.15   270023     0.00     0.00  sweep_slice


Ideas? I'm currently using mldonkey-3.1.3. Going back to older versions didn't help. I think this issue started happening around the beginning of August.
Back to top
View user's profile Send private message
ygrek
professional


Joined: 20 Mar 2010
Posts: 593

PostPosted: Thu Oct 10, 2013 5:49 am    Post subject: Reply with quote

This is interesting. But gprof is not really useful, please profile it with http://poormansprofiler.org/ - here is the pmp script I use : http://ygrek.org.ua/p/code/pmp
e.g. run pmp -i 100 -p $(pgrep mlnet)
Also please show sysinfo output
_________________
Download | Report bugs | git mirror
Back to top
View user's profile Send private message Visit poster's website
dennisn
user


Joined: 08 Sep 2004
Posts: 81
Location: Canada

PostPosted: Sun Oct 13, 2013 12:51 pm    Post subject: Reply with quote

Poor man really appreciated that profiler! So where is the bottleneck? :p Here are the top portions of a few runs:

Code:
    200 __kernel_vsyscall,pthread_cond_timedwait,hasher_thread,start_thread,clone
     60 __kernel_vsyscall,poll,try_poll,ml_select,camlBasicSocket__loop_1558,::??
      7 __kernel_vsyscall,poll,try_poll,ml_select,camlBasicSocket__loop_1558,camlBasicSocket__77,::??
      4 camlBasicSocket__iter_task_1504,::??,camlBasicSocket__loop_1558,camlBasicSocket__loop_1558,camlCommonMain,::??
      3 libc::??,mcount,caml_apply2,camlCommonFile__219,::??,camlQueue__fold_1087,::??
      3 camlQueue__fold_1087,::??
      2 __kernel_vsyscall,__read_chk,unix_read,camlUnix__read_1237,::??,::??,camlUnix32__read_1137,camlUnix32__read_1137,::??
      1 try_poll,ml_select,camlBasicSocket__loop_1558,camlBasicSocket__77,::??
      1 mcount,caml_apply2,camlCommonFile__219,::??,camlQueue__fold_1087,::??
      1 libc::??,mcount,caml_modify,camlBasicSocket__iter_task_1504,::??,camlBasicSocket__loop_1558,
       camlBasicSocket__loop_1558,camlCommonMain,::??


    200 __kernel_vsyscall,pthread_cond_timedwait,hasher_thread,start_thread,clone
     56 __kernel_vsyscall,poll,try_poll,ml_select,camlBasicSocket__loop_1558,::??
     21 __kernel_vsyscall,poll,try_poll,ml_select,camlBasicSocket__loop_1558,camlBasicSocket__77,::??
      4 libc::??,mcount,camlCommonFile__fun_2602,camlCommonFile__219,::??,camlQueue__fold_1087,::??
      2 libc::??,mcount,caml_modify,camlBasicSocket__iter_task_1504,::??,camlBasicSocket__loop_1558,
       camlBasicSocket__loop_1558,camlCommonMain,::??
      2 libc::??,mcount,camlDonkeyGlobals__fun_3141,::??,camlCommonSources__fun_2543,::??
      1 try_poll,ml_select,camlBasicSocket__loop_1558,camlBasicSocket__77,::??
      1 mark_slice,caml_major_collection_slice,caml_minor_collection,caml_garbage_collection,caml_call_gc,
       camlBasicSocket__loop_1558,camlBasicSocket__loop_1558,camlCommonMain,::??


    200 __kernel_vsyscall,pthread_cond_timedwait,hasher_thread,start_thread,clone
     57 __kernel_vsyscall,poll,try_poll,ml_select,camlBasicSocket__loop_1558,::??
      7 __kernel_vsyscall,poll,try_poll,ml_select,camlBasicSocket__loop_1558,camlBasicSocket__77,::??
      2 libc::??,mcount,caml_greaterequal,camlPervasives__max_1025,::??,::??,::??,camlBig_int__quomod_big_int_1118,::??
      2 libc::??,mcount,camlCommonFile__fun_2602,camlCommonFile__219,::??,camlQueue__fold_1087,::??
      2 libc::??,mcount,caml_c_call,camlBig_int__add_big_int_1086,::??,camlKademlia__loop_1172,::??
      2 __kernel_vsyscall,__read_chk,unix_read,camlUnix__read_1237,::??,::??,camlUnix32__read_1137,camlUnix32__read_1137,::??
      2 camlBasicSocket__iter_task_1504,camlBasicSocket__iter_task_1504,::??,::??,camlBasicSocket__loop_1558,
       camlBasicSocket__loop_1558,camlCommonMain,::??
      1 sweep_slice,caml_major_collection_slice,caml_minor_collection,caml_garbage_collection,caml_call_gc,
       camlCommonSwarming__iter_3444,::??,::??
      1 mcount,caml_fl_allocate,caml_alloc_shr,caml_oldify_one,caml_empty_minor_heap,caml_minor_collection,
       caml_garbage_collection,caml_call_gc,camlBasicSocket__loop_1558,camlBasicSocket__loop_1558,camlCommonMain,::??
      1 mcount,camlCommonFile__fun_2602,camlCommonFile__219,::??,camlQueue__fold_1087,::??
      1 mcount,caml_apply2,camlCommonFile__219,::??,camlQueue__fold_1087,::??



Here is my sysinfo:

Code:
Buildinfo Version: MLNet Multi-Network p2p client version 3.1.3

Networks: Global Shares Direct Connect FileTP BitTorrent Donkey (SUI)

OCaml version: 3.12.0 - C compiler version: 4.4.5 - C++ compiler version: 4.4.5
Built on: Linux i686 3.0.1 (little endian) with glibc 2.12.1
Configure args: '--enable-option-checking=fatal' '--prefix=/usr' '--build=i686-pc-linux-gnu' '--host=i686-pc-linux-gnu' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--datadir=/usr/share' '--sysconfdir=/etc' '--localstatedir=/var/lib' 'LIBS=' '--sysconfdir=/etc/mldonkey' '--sharedstatedir=/var/mldonkey' '--localstatedir=/var/mldonkey' '--enable-checks' '--enable-profile' '--enable-debug' '--disable-batch' '--enable-bittorrent' '--disable-fasttrack' '--disable-gnutella' '--disable-gnutella2' '--disable-gd' '--disable-magic' '--disable-gui' 'build_alias=i686-pc-linux-gnu' 'host_alias=i686-pc-linux-gnu' 'CFLAGS=-pg' 'LDFLAGS=-Wl,-O1 -Wl,--as-needed' 'CXXFLAGS=-O2 -march=i686 -pipe'

Features: threads zlib-1.2.3 bzip2-1.0.6 no-gd iconv(active) no-magic no-upnp no-natpmp check-bounds

Enabled nets: Donkey BitTorrent
Server usage: enabled
Geoip:  enabled, GeoLite data created by MaxMind, available from http://maxmind.com/
IP blocking: local: 0 ranges - web: 206793 ranges
System info: Linux 3.0.1 #2 Sat Aug 13 17:47:42 EDT 2011 i686
   language: EN - locale: UTF-8 - UTC offset: -0400
   max_string_length: 16777211 - word_size: 32 - max_array_length: 4194303 - max_int: 1073741823
   max file descriptors: 4096 - max useable file size: 2^63-1 bits (do the maths ;-p)




Here is my meminfo:

Code:
 Gc
 minor_words: 31859393296
 promoted_words: 4288255624
 major_words: 5911627814
 minor_collections: 981946
 major_collections: 2563
 heap_words: 11120640
 heap_chunks: 69
 compactions: 29
 top_heap_words: 15180800
 allocated_bytes: 133931061944
 Unix32
   table: 388
   modified_files: 0
   max cache_size: 2941
   fd cache_size: 8
 BasicSocket
   before_select_hooks: 2
   after_select_hooks: 1
   bandwidth_second_timers: 2
   42 timers
   2626 fd_tasks:
   0 closed_tasks:
 Ip
   22 IPs in ip_cache
   0 entries in hostname_table
   0 entries in ip_fifo
 Ip_set
   descriptions: 1
 GeoIp
   countries: 253
   database_type: country edition
   map size: 587854
 tcpBufferedSocket
   26 latencies
   String.length big_buffer: 65536
   connection_managers: 6
   read_bandwidth_controlers: 1
   write_bandwidth_controlers: 1
   to_deflate: 0
   max_opened_connections: 110
   max_connections_per_second: 5
   max_buffer_size: 500000
 udpSocket
   8 latencies
   0 entries in pings_fifo
   0 entries in pings_hashtbl
 CommonUserDb
   grouplist: 2
   userlist: 2
 CommonGlobals
   intern_table: 2
  core_gui_fifo: 0
  gui_core_fifo: 0
  chat_message_fifo: 0
  upload_history: 721
  download_history: 721
  upload_h_history: 17
  download_h_history: 17
  bandwidth_samples: 20
  short_delay_bandwidth_samples: 5
  dummy_sample: 6
  activities: 1019
 CommonUser
   users: 0
 CommonNetwork
   networks_by_name: 5
   networks_by_num: 5
   commands_by_kind: 17
 CommonServer
   servers: 12
 CommonClient
   clients: 1595
   uploaders: 9
 CommonFile
   files: 403
   files_ops: 5
 CommonResult
   results_by_num: 0
   results_by_uid: 0
   known_uids: 0
   store: 0 loaded/ 0 max
 CommonBlocking
   local ranges: 0
   web ranges: 206793
 CommonComplexOptions
   friends: 1
   contacts: 0
 CommonShared
   shared: 401
   dirnames: 3
 CommonRoom
   rooms: 1
 CommonSearch
   Filtering index memory: 11
   Local index memory: 11
   Memorized searches: 0
   Memorized items: 0
 CommonSwarming
   Swarmers: 17
     nchunks: 14527 nblocks: 43 nranges: 134
   Storage (without blocks): 262803 bytes
   Uploaders: 19
   Storage: 50132 bytes
 CommonUploads
   infos_by_name: 0
   shareds_by_uid: 0
   shareds_by_id: 0
   shared_files: 0
   pending_slots: 1
 CommonStats
   countries: 224
 FileTPGlobals
   current_files: 0
   file_by_uid: 0
   clients_by_uid: 0
   protos_by_name: 3
 BittorrentGlobals
 Number of old files: 2134
 Files in downloads directory: 19
 Files in tracked directory: 0
 Files in seeded directory: 22
 files_by_uid: 38
 ft_by_num: 0
 BittorrentDHT
 Routing : 247 nodes in 31 buckets
 Storage : 0 keys with 0 peers
 rpc recv 65040 pkts (15786676 bytes)
 rpc sent 160980 pkts (16531505 bytes)
 rpc decoded 64962, handled 64947
 rpc timeouted 96084, orphan 105
 Outgoing queries : ok/error/timeout
 PING: 11848/0/5502
 FIND_NODE: 45725/10/81059
 GET_PEERS: 3776/0/8752
 ANNOUNCE: 2853/152/771
 Incoming queries
 PING: 123
 FIND_NODE: 196
 GET_PEERS: 166
 ANNOUNCE: 8
 DonkeySources
 For all managers (362):
    Queue[new_sources]: 0 entries (0 ready)
    Queue[good_sources]: 0 entries (0 ready)
    Queue[ready_saved_sources]: 0 entries (0 ready)
    Queue[waiting_saved_sources]: 0 entries (0 ready)
    Queue[old_sources1]: 0 entries (0 ready)
    Queue[old_sources2]: 0 entries (0 ready)
    Queue[old_sources3]: 2 entries (0 ready)
    Queue[do_not_try_queue]: 0 entries (0 ready)
    Queue[connected_sources]: 8 entries (8 ready)
    Queue[connecting_sources]: 0 entries (0 ready)
    Queue[busy_sources]: 0 entries (0 ready)
 Sources by UID table: 11 entries
 Sources by UID table stats: 13557 11 240 0 0 3
 Sources by NUM table: 11 entries
 Sources by NUM table stats: 13557 11 1107 0 0 3
 Used indirect connections: 7
 Connecting Sources: 0 entries
 Next Direct Sources: 0 entries
 Next Indirect Sources: 0 entries
 DonkeyGlobals
 Clients: 10
    Bad Clients: 0/1
    Read Buffers: 0
    Write Messages: 500000
    Uninteresting clients: 0
    Connected clients: 7
    Aliased clients: 0
    Closed clients: 0
    Unlocated clients: 0
    Bad numbered clients: 0
    Dead clients: 0
    Disconnected aliases: 0
 Number of old files: 0
 Current files: 1
   udp_clients: 0
   client_to_client_tags: 8
   client_to_server_tags: 4
   overnet_connectreply_tags: 2
   overnet_connect_tags: 2
   clients_root: 2
   servers_list: 9
   xs_servers_list: 0
   connected_server_list: 1
   connecting_server_list: 0
   udp_servers_list: 0
   interesting_clients: 0
   shared_files: 0
   servers_by_key: 12
   banned_ips: 0
   old_requests: 34
   connected_clients: 7
   files_by_md4: 362
   shared_files_info: 364
   file_groups: 0
   udp_servers_replies: 0
   join_queue_by_md4: 0
   join_queue_by_id: 0
Back to top
View user's profile Send private message
ygrek
professional


Joined: 20 Mar 2010
Posts: 593

PostPosted: Sun Oct 13, 2013 3:50 pm    Post subject: Reply with quote

Doesn't look like any particular cpu-intensive tasks. Try raising loop_delay..
PS Also, looks like you are still running profiling-enabled build - it is not needed for pmp.
PPS Also try building with the newer OCaml version - it is constantly improving wrt generated code speed
_________________
Download | Report bugs | git mirror
Back to top
View user's profile Send private message Visit poster's website
dennisn
user


Joined: 08 Sep 2004
Posts: 81
Location: Canada

PostPosted: Sat Oct 19, 2013 2:05 am    Post subject: Reply with quote

Progress report: Upgrading to ocaml-4.01 did not help. Increasing loop_delay (to 50, 100) only kinda helped -- the cpu-usage graph still continued to rise, but slower than before .. perhaps half as steep as before. After a loop_delay of 100 overnight, I awoke to an unresponsive mlnet, and a 30MiB mlnet.log file flooded with:

Code:
(bS) accept failed: Too many open files : unexpected exn exec can_read
Back to top
View user's profile Send private message
dennisn
user


Joined: 08 Sep 2004
Posts: 81
Location: Canada

PostPosted: Sat Oct 19, 2013 1:36 pm    Post subject: Reply with quote

Looking at my /proc/$(pidof mlnet)/fd folder indeed shows an increasing number of open files... after an hour or so of running, the folder already has over 300, almost all broken symlinks like "302 -> socket:[64293748]". Sometimes the number of them decreases a bit, but the general trend is up. (My system currently has a limit of 4000 per process :s). Their timestamps don't seem evenly distributed. There a a huge cluster of over 200 of them on one minute, and gaps of a few minutes between the others.

Someone isn't closing file handles? Razz
Back to top
View user's profile Send private message
dennisn
user


Joined: 08 Sep 2004
Posts: 81
Location: Canada

PostPosted: Sat Oct 19, 2013 5:32 pm    Post subject: Reply with quote

Probably related, in my mlnet.log, I do have many "close failed" messages, like I mentioned before:

Code:
 [HTTPsv] 217.118.83.186:38454 blocked: Unknown
 [HTTPsv] 217.118.83.186:38454 blocked: Unknown
 [HTTPsv] connection from 217.118.83.186 rejected (IP is blocked)
 [bS] close failed: Bad file descriptor : unexpected exn exec can_read

After about 5 hours of uptime, I have about 120 such messages in the log file, but over 1800 broken symlinks in mlnet's file-descripter fd/ folder.
Back to top
View user's profile Send private message
dennisn
user


Joined: 08 Sep 2004
Posts: 81
Location: Canada

PostPosted: Sat Oct 19, 2013 6:43 pm    Post subject: Reply with quote

The number of "fd_tasks" listed in mem_stats is the same as the number of broken fd sockets in my /proc//fd folder. Currently, after restarting and running for barely an hour, I have about 500.

Code:
 BasicSocket                                         
   before_select_hooks: 2                             
   after_select_hooks: 1                             
   bandwidth_second_timers: 2                         
   42 timers                                         
   496 fd_tasks:                                     
   0 closed_tasks:
Back to top
View user's profile Send private message
dennisn
user


Joined: 08 Sep 2004
Posts: 81
Location: Canada

PostPosted: Fri Nov 15, 2013 1:45 am    Post subject: Reply with quote

So, any idea why I have so many open sockets? (fd_task's) Or what those mlnet.log messages mean, about "close failed"s -- why did close attempts fail? And is the edonkey network being spammed/attacked? Or just me? :p What does the "HTTPsv" tag mean?

All those open connections vastly exceed my "max_opened_connections" default of 200 -- how is that even possible? "debug_socks" shows thousands of "FD http connection"s, and a handful of "FD (bittorrent|donkey|gui|etc) connection". The http connections have lifetimes that range from 0seconds to the lifetime of my mlnet process -- i.e. they're steadily being created.

Here is a sample of one of my "debug_socks":
http://dennisn.dyndns.org/guest/pubstuff/mlnet-debug/mlnet-debug_socks.txt

This is a critical bug for me.
Back to top
View user's profile Send private message
ygrek
professional


Joined: 20 Mar 2010
Posts: 593

PostPosted: Fri Nov 15, 2013 6:23 pm    Post subject: Reply with quote

As you have figured before someone is not closing sockets, hence the fd leak.. IIUC there are less "close failed" messages than open sockets, so it is not strict relation? The problem looks like in http connections (i.e. the ones that are used for web interface) and they are not limited by max_opened_connections.. Doesn't look like problem with edk.. Do you have any hint when it started? As a workaround - try "close_all_sockets" command - does it close all those hanging fd_tasks?

UPDATE: I can reproduce those pesky '[bS] close failed: Bad file descriptor : unexpected exn exec can_read' messages, it doesn't result in leaked fds, so it is not directly related.
Can it be possible that somebody accesses web interface and doesn't close connections (I can hardly imagine how that can happen....)
_________________
Download | Report bugs | git mirror
Back to top
View user's profile Send private message Visit poster's website
dennisn
user


Joined: 08 Sep 2004
Posts: 81
Location: Canada

PostPosted: Sat Nov 16, 2013 1:00 am    Post subject: Reply with quote

"close_all_sockets" doesn't change anything. "netstat -pnat" shows that the connections are all (non-blocked) global IPs with ESTABLISHED connections to my global IP on port 6881, which is the default "BT-tracker_port".
Back to top
View user's profile Send private message
dennisn
user


Joined: 08 Sep 2004
Posts: 81
Location: Canada

PostPosted: Sat Nov 16, 2013 4:58 am    Post subject: Reply with quote

Setting the tracker port to 0 (disabling it) seems to be a decent bandaid solution. What are the advantages/disadvantages to disabling this feature?
Back to top
View user's profile Send private message
ygrek
professional


Joined: 20 Mar 2010
Posts: 593

PostPosted: Sat Nov 16, 2013 6:46 am    Post subject: Reply with quote

You don't need the tracker at all unless you are distributing your own torrents with the address of this tracker embedded into those torrent files.
_________________
Download | Report bugs | git mirror
Back to top
View user's profile Send private message Visit poster's website
Display posts from previous:   
Post new topic   Reply to topic    MLDonkey Forum Index -> Problems with MLDonkey Client (in general, and ed2k, kad, overnet) All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Sourceforge.net Logo