-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Performance history
Andres Riancho edited this page Sep 2, 2014
·
9 revisions
This page holds historical performance information captured by collector and analyzed by w3af-performance-analysis.
The raw data is stored in a private S3 bucket named w3af-performance-data. If you're interested in getting that information, please ask Andrés for access.
Increased the run time for w3af to 15min
Performance analysis
====================
Total memory size
0: 32.4MiB
1: 93.1MiB
2: 107.0MiB
3: 107.0MiB
Memory usage summary
First:
Total 201133 objects, 516 types, Total size = 32.4MiB (33995261 bytes)
Index Count % Size % Cum Max Kind
0 7500 3 8082336 23 23 196888 dict
1 61171 30 5746150 16 40 12467 str
2 1324 0 3303360 9 50 49488 module
3 37725 18 2647940 7 58 9236 unicode
4 18522 9 2370816 6 65 128 code
5 18943 9 2273160 6 71 120 function
6 2479 1 2241016 6 78 904 type
7 13947 6 1059536 3 81 3184 tuple
8 7509 3 854720 2 84 20192 list
9 524 0 468704 1 85 8424 set
10 1024 0 360448 1 86 352 U32
11 356 0 321824 0 87 904 Packet_metaclass
12 3423 1 301224 0 88 88 weakref
13 8685 4 208440 0 88 24 int
14 165 0 176736 0 89 1120 ByteEnumField
15 376 0 170648 0 89 12584 OrderedDict
16 122 0 133184 0 90 1120 BitField
17 110 0 122320 0 90 1112 Distribution
18 108 0 118272 0 91 1120 ShortEnumField
19 1428 0 114240 0 91 80 wrapper_descriptor
Before last:
Total 363806 objects, 569 types, Total size = 107.0MiB (112161943 bytes)
Index Count % Size % Cum Max Kind
0 79415 21 44760424 39 39 1624292 unicode
1 36940 10 17036704 15 55 196888 dict
2 9923 2 11026696 9 64 1112 URL
3 80320 22 10540180 9 74 128263 str
4 1325 0 3306768 2 77 49488 module
5 9635 2 2851960 2 79 296 QueryString
6 27479 7 2633232 2 82 20192 list
7 18532 5 2372096 2 84 128 code
8 18985 5 2278200 2 86 120 function
9 2485 0 2246440 2 88 904 type
10 26208 7 1961288 1 90 3184 tuple
11 753 0 1262696 1 91 8424 set
12 827 0 901192 0 91 1112 FuzzableRequest
13 997 0 457928 0 92 1064 Headers
14 4557 1 437472 0 92 96 lxml.etree._LogEntry
15 1024 0 360448 0 93 352 U32
16 1002 0 348528 0 93 1112 _Condition
17 356 0 321824 0 93 904 Packet_metaclass
18 265 0 313952 0 93 3424 HTTPResponse
19 3521 0 309848 0 94 88 weakref
Operating System information
Load average: 8.69 7.94 5.37
Network
Bytes sent: 17.1M
Bytes received: 337.7M
Swap memory
% used (best: 0.0): 0.0
Pages per second (out) (best: 0): 0
Pages per second (in) (best: 0): 0
CPU usage by function
Measurement #0 (w3af-10274-2014-02-09-02_20.cpu)
0: cpu_usage.py:56(dump_data)
1: utils.py:34(dump_data_every_thread)
Measurement #1 (w3af-10274-2014-02-09-02_22.cpu)
0: threadpool.py:42(one_to_many.__call__)
1: web_spider.py:345(web_spider._verify_reference)
2: extended_urllib.py:434(ExtendedUrllib._send)
3: pool.py:64(mapstar)
4: extended_urllib.py:252(ExtendedUrllib.GET)
5: plugin.py:217(meth)
6: urllib2.py:386(OpenerDirector.open)
7: utils.py:34(dump_data_every_thread)
8: memory_usage.py:33(inner)
9: memory_usage.py:42(start_memory_profiling)
10: memory_usage.py:53(dump_objects)
11: scanner.py:123(dump_all_objects)
12: pstats.py:451(TupleComp)
13: web_spider.py:170(web_spider._urls_to_verify_generator)
14: pool.py:298(Pool.map_async)
15: url.py:173(URL.__init__)
16: base_consumer.py:42(crawl_infrastructure._wrapper)
17: urllib2.py:374(OpenerDirector._call_chain)
18: urllib2.py:414(OpenerDirector._open)
19: HTTPResponse.py:127(from_httplib_resp)
Measurement #2 (w3af-10274-2014-02-09-02_24.cpu)
0: extended_urllib.py:434(ExtendedUrllib._send)
1: extended_urllib.py:252(ExtendedUrllib.GET)
2: plugin.py:217(meth)
3: web_spider.py:345(web_spider._verify_reference)
4: threadpool.py:42(one_to_many.__call__)
5: pool.py:64(mapstar)
6: urllib2.py:386(OpenerDirector.open)
7: base_consumer.py:42(crawl_infrastructure._wrapper)
8: HTTPResponse.py:127(from_httplib_resp)
9: url.py:173(URL.__init__)
10: urllib2.py:374(OpenerDirector._call_chain)
11: urllib2.py:414(OpenerDirector._open)
12: headers.py:34(Headers.__init__)
13: web_spider.py:170(web_spider._urls_to_verify_generator)
14: url.py:47(URL.changed_wrapper)
15: pool.py:298(Pool.map_async)
16: threadpool.py:54(return_args.__call__)
17: threadpool.py:78(Pool.map_multi_args)
18: extended_urllib.py:205(ExtendedUrllib.send_mutant)
19: utils.py:34(dump_data_every_thread)
Measurement #3 (w3af-10274-2014-02-09-02_26.cpu)
0: extended_urllib.py:434(ExtendedUrllib._send)
1: extended_urllib.py:252(ExtendedUrllib.GET)
2: plugin.py:217(meth)
3: pool.py:64(mapstar)
4: threadpool.py:42(one_to_many.__call__)
5: web_spider.py:345(web_spider._verify_reference)
6: urllib2.py:386(OpenerDirector.open)
7: base_consumer.py:42(crawl_infrastructure._wrapper)
8: web_spider.py:170(web_spider._urls_to_verify_generator)
9: utils.py:34(dump_data_every_thread)
10: url.py:173(URL.__init__)
11: HTTPResponse.py:127(from_httplib_resp)
12: memory_usage.py:53(dump_objects)
13: scanner.py:123(dump_all_objects)
14: pstats.py:451(TupleComp)
15: headers.py:34(Headers.__init__)
16: threadpool.py:78(Pool.map_multi_args)
17: urllib2.py:414(OpenerDirector._open)
18: urllib2.py:374(OpenerDirector._call_chain)
19: pool.py:298(Pool.map_async)
Measurement #4 (w3af-10274-2014-02-09-02_29.cpu)
0: web_spider.py:345(web_spider._verify_reference)
1: plugin.py:217(meth)
2: extended_urllib.py:434(ExtendedUrllib._send)
3: pool.py:64(mapstar)
4: threadpool.py:42(one_to_many.__call__)
5: extended_urllib.py:252(ExtendedUrllib.GET)
6: urllib2.py:386(OpenerDirector.open)
7: base_consumer.py:42(crawl_infrastructure._wrapper)
8: web_spider.py:170(web_spider._urls_to_verify_generator)
9: url.py:173(URL.__init__)
10: threadpool.py:78(Pool.map_multi_args)
11: pool.py:298(Pool.map_async)
12: HTTPResponse.py:127(from_httplib_resp)
13: threadpool.py:54(return_args.__call__)
14: url.py:47(URL.changed_wrapper)
15: utils.py:34(dump_data_every_thread)
16: urllib2.py:414(OpenerDirector._open)
17: crawl_plugin.py:41(web_spider.crawl_wrapper)
18: urllib2.py:374(OpenerDirector._call_chain)
19: headers.py:34(Headers.__init__)
Measurement #5 (w3af-10274-2014-02-09-02_31.cpu)
0: web_spider.py:345(web_spider._verify_reference)
1: extended_urllib.py:434(ExtendedUrllib._send)
2: plugin.py:217(meth)
3: threadpool.py:42(one_to_many.__call__)
4: pool.py:64(mapstar)
5: extended_urllib.py:252(ExtendedUrllib.GET)
6: urllib2.py:386(OpenerDirector.open)
7: base_consumer.py:42(crawl_infrastructure._wrapper)
8: web_spider.py:170(web_spider._urls_to_verify_generator)
9: pool.py:298(Pool.map_async)
10: threadpool.py:78(Pool.map_multi_args)
11: url.py:173(URL.__init__)
12: HTTPResponse.py:127(from_httplib_resp)
13: parser_cache.py:51(ParserCache.get_document_parser_for)
14: url.py:47(URL.changed_wrapper)
15: urllib2.py:414(OpenerDirector._open)
16: urllib2.py:374(OpenerDirector._call_chain)
17: threadpool.py:54(return_args.__call__)
18: document_parser.py:40(DocumentParser.__init__)
19: html.py:45(HTMLParser.__init__)
Measurement #6 (w3af-10274-2014-02-09-02_33.cpu)
0: web_spider.py:345(web_spider._verify_reference)
1: extended_urllib.py:434(ExtendedUrllib._send)
2: plugin.py:217(meth)
3: threadpool.py:42(one_to_many.__call__)
4: pool.py:64(mapstar)
5: extended_urllib.py:252(ExtendedUrllib.GET)
6: urllib2.py:386(OpenerDirector.open)
7: web_spider.py:170(web_spider._urls_to_verify_generator)
8: base_consumer.py:42(crawl_infrastructure._wrapper)
9: pool.py:298(Pool.map_async)
10: threadpool.py:78(Pool.map_multi_args)
11: url.py:173(URL.__init__)
12: HTTPResponse.py:127(from_httplib_resp)
13: parser_cache.py:51(ParserCache.get_document_parser_for)
14: web_spider.py:270(web_spider._should_output_extracted_url)
15: url.py:47(URL.changed_wrapper)
16: threadpool.py:54(return_args.__call__)
17: sgml.py:137(HTMLParser._parse)
18: urllib2.py:414(OpenerDirector._open)
19: extended_urllib.py:205(ExtendedUrllib.send_mutant)
Core status summary
Measurement #0 (w3af-10274-2014-02-09-02_22.core)
Audit queue input speed: 126.49
Requests per minute: 456
Audit queue output speed: 67.04
Crawl queue size: 0
Crawl queue input speed: 126.49
Crawl queue output speed: 126.51
Audit queue size: 91
Requests sent: 913
Measurement #1 (w3af-10274-2014-02-09-02_24.core)
Audit queue input speed: 92.49
Requests per minute: 441
Audit queue output speed: 47.84
Crawl queue size: 0
Crawl queue input speed: 92.49
Crawl queue output speed: 142.12
Audit queue size: 166
Requests sent: 1768
Measurement #2 (w3af-10274-2014-02-09-02_26.core)
Audit queue input speed: 71.84
Requests per minute: 401
Audit queue output speed: 53.86
Crawl queue size: 0
Crawl queue input speed: 71.84
Crawl queue output speed: 72.71
Audit queue size: 237
Requests sent: 2412
Measurement #3 (w3af-10274-2014-02-09-02_28.core)
Audit queue input speed: 77.13
Requests per minute: 369
Audit queue output speed: 54.37
Crawl queue size: 0
Crawl queue input speed: 77.13
Crawl queue output speed: 77.21
Audit queue size: 313
Requests sent: 2958
Measurement #4 (w3af-10274-2014-02-09-02_30.core)
Audit queue input speed: 82.41
Requests per minute: 349
Audit queue output speed: 56.66
Crawl queue size: 0
Crawl queue input speed: 82.41
Crawl queue output speed: 82.43
Audit queue size: 365
Requests sent: 3500
Measurement #5 (w3af-10274-2014-02-09-02_32.core)
Audit queue input speed: 104.12
Requests per minute: 344
Audit queue output speed: 37.3
Crawl queue size: 5
Crawl queue input speed: 104.12
Crawl queue output speed: 99.68
Audit queue size: 484
Requests sent: 4140
Measurement #6 (w3af-10274-2014-02-09-02_34.core)
Audit queue input speed: 127.03
Requests per minute: 341
Audit queue output speed: 53.39
Crawl queue size: 67
Crawl queue input speed: 127.03
Crawl queue output speed: 70.7
Audit queue size: 602
Requests sent: 4790
Performance analysis
====================
Total memory size
0: 32.5MiB
1: 96.7MiB
2: 162.9MiB
Memory usage summary
First:
Total 201618 objects, 517 types, Total size = 32.5MiB (34088982 bytes)
Index Count % Size % Cum Max Kind
0 7509 3 8084856 23 23 196888 dict
1 61246 30 5749799 16 40 12467 str
2 1324 0 3303360 9 50 49488 module
3 37725 18 2647940 7 58 9236 unicode
4 18522 9 2370816 6 64 128 code
5 18943 9 2273160 6 71 120 function
6 2479 1 2241016 6 78 904 type
7 14056 6 1070208 3 81 3184 tuple
8 7582 3 860008 2 83 20192 list
9 524 0 468704 1 85 8424 set
10 1024 0 360448 1 86 352 U32
11 356 0 321824 0 87 904 Packet_metaclass
12 3423 1 301224 0 88 88 weakref
13 8685 4 208440 0 88 24 int
14 165 0 176736 0 89 1120 ByteEnumField
15 376 0 170648 0 89 12584 OrderedDict
16 122 0 133184 0 90 1120 BitField
17 110 0 122320 0 90 1112 Distribution
18 108 0 118272 0 90 1120 ShortEnumField
19 1428 0 114240 0 91 80 wrapper_descriptor
Before last:
Total 299549 objects, 566 types, Total size = 96.7MiB (101409775 bytes)
Index Count % Size % Cum Max Kind
0 63241 21 43581956 42 42 735540 unicode
1 72958 24 14341688 14 57 488383 str
2 24376 8 13129408 12 70 196888 dict
3 5359 1 5934632 5 75 1112 URL
4 1325 0 3306768 3 79 49488 module
5 18529 6 2371712 2 81 128 code
6 18987 6 2278440 2 83 120 function
7 2485 0 2246440 2 85 904 type
8 25786 8 1934704 1 87 3184 tuple
9 17390 5 1744632 1 89 20192 list
10 5261 1 1557256 1 91 296 QueryString
11 660 0 835616 0 91 8424 set
12 400 0 429824 0 92 1112 FuzzableRequest
13 672 0 413184 0 92 1064 Headers
14 303 0 360840 0 93 3424 HTTPResponse
15 1024 0 360448 0 93 352 U32
16 356 0 321824 0 93 904 Packet_metaclass
17 639 0 312960 0 94 624 frame
18 3521 1 309848 0 94 88 weakref
19 217 0 237464 0 94 1112 HTTPRequest
Operating System information
Load average: 5.92 7.26 5.46
Network
Bytes sent: 15.7M
Bytes received: 328.3M
Swap memory
% used (best: 0.0): 0.0
Pages per second (out) (best: 0): 0
Pages per second (in) (best: 0): 4096
CPU usage by function
Measurement #0 (w3af-10273-2014-02-09-02_20.cpu)
0: cpu_usage.py:56(dump_data)
1: utils.py:34(dump_data_every_thread)
Measurement #1 (w3af-10273-2014-02-09-02_22.cpu)
0: extended_urllib.py:434(ExtendedUrllib._send)
1: extended_urllib.py:252(ExtendedUrllib.GET)
2: threadpool.py:42(one_to_many.__call__)
3: web_spider.py:345(web_spider._verify_reference)
4: plugin.py:217(meth)
5: pool.py:64(mapstar)
6: urllib2.py:386(OpenerDirector.open)
7: utils.py:34(dump_data_every_thread)
8: memory_usage.py:33(inner)
9: memory_usage.py:42(start_memory_profiling)
10: memory_usage.py:53(dump_objects)
11: scanner.py:123(dump_all_objects)
12: pstats.py:451(TupleComp)
13: base_consumer.py:42(crawl_infrastructure._wrapper)
14: HTTPResponse.py:127(from_httplib_resp)
15: url.py:173(URL.__init__)
16: web_spider.py:170(web_spider._urls_to_verify_generator)
17: urllib2.py:414(OpenerDirector._open)
18: urllib2.py:374(OpenerDirector._call_chain)
19: url.py:47(URL.changed_wrapper)
Measurement #2 (w3af-10273-2014-02-09-02_24.cpu)
0: extended_urllib.py:434(ExtendedUrllib._send)
1: extended_urllib.py:252(ExtendedUrllib.GET)
2: plugin.py:217(meth)
3: web_spider.py:345(web_spider._verify_reference)
4: urllib2.py:386(OpenerDirector.open)
5: threadpool.py:42(one_to_many.__call__)
6: pool.py:64(mapstar)
7: HTTPResponse.py:127(from_httplib_resp)
8: url.py:173(URL.__init__)
9: base_consumer.py:42(crawl_infrastructure._wrapper)
10: headers.py:34(Headers.__init__)
11: urllib2.py:414(OpenerDirector._open)
12: urllib2.py:374(OpenerDirector._call_chain)
13: url.py:47(URL.changed_wrapper)
14: utils.py:34(dump_data_every_thread)
15: db.py:67(store_in_cache)
16: memory_usage.py:33(inner)
17: memory_usage.py:42(start_memory_profiling)
18: memory_usage.py:53(dump_objects)
19: scanner.py:123(dump_all_objects)
Measurement #3 (w3af-10273-2014-02-09-02_27.cpu)
0: extended_urllib.py:434(ExtendedUrllib._send)
1: extended_urllib.py:252(ExtendedUrllib.GET)
2: plugin.py:217(meth)
3: urllib2.py:386(OpenerDirector.open)
4: pool.py:64(mapstar)
5: threadpool.py:42(one_to_many.__call__)
6: web_spider.py:345(web_spider._verify_reference)
7: HTTPResponse.py:127(from_httplib_resp)
8: urllib2.py:414(OpenerDirector._open)
9: urllib2.py:374(OpenerDirector._call_chain)
10: url.py:173(URL.__init__)
11: headers.py:34(Headers.__init__)
12: url.py:47(URL.changed_wrapper)
13: web_spider.py:170(web_spider._urls_to_verify_generator)
14: base_consumer.py:42(crawl_infrastructure._wrapper)
15: extended_urllib.py:205(ExtendedUrllib.send_mutant)
16: db.py:67(store_in_cache)
17: cache.py:78(CacheHandler.http_response)
18: threadpool.py:54(return_args.__call__)
19: url.py:306(URL.set_querystring)
Measurement #4 (w3af-10273-2014-02-09-02_29.cpu)
0: extended_urllib.py:434(ExtendedUrllib._send)
1: extended_urllib.py:252(ExtendedUrllib.GET)
2: plugin.py:217(meth)
3: urllib2.py:386(OpenerDirector.open)
4: web_spider.py:345(web_spider._verify_reference)
5: pool.py:64(mapstar)
6: threadpool.py:42(one_to_many.__call__)
7: url.py:173(URL.__init__)
8: HTTPResponse.py:127(from_httplib_resp)
9: urllib2.py:374(OpenerDirector._call_chain)
10: web_spider.py:170(web_spider._urls_to_verify_generator)
11: urllib2.py:414(OpenerDirector._open)
12: url.py:47(URL.changed_wrapper)
13: headers.py:34(Headers.__init__)
14: base_consumer.py:42(crawl_infrastructure._wrapper)
15: extended_urllib.py:205(ExtendedUrllib.send_mutant)
16: parser_cache.py:51(ParserCache.get_document_parser_for)
17: threadpool.py:54(return_args.__call__)
18: document_parser.py:40(DocumentParser.__init__)
19: html.py:45(HTMLParser.__init__)
Measurement #5 (w3af-10273-2014-02-09-02_31.cpu)
0: extended_urllib.py:434(ExtendedUrllib._send)
1: plugin.py:217(meth)
2: extended_urllib.py:252(ExtendedUrllib.GET)
3: urllib2.py:386(OpenerDirector.open)
4: web_spider.py:345(web_spider._verify_reference)
5: threadpool.py:42(one_to_many.__call__)
6: pool.py:64(mapstar)
7: url.py:173(URL.__init__)
8: base_consumer.py:42(crawl_infrastructure._wrapper)
9: web_spider.py:170(web_spider._urls_to_verify_generator)
10: HTTPResponse.py:127(from_httplib_resp)
11: pool.py:298(Pool.map_async)
12: url.py:47(URL.changed_wrapper)
13: urllib2.py:374(OpenerDirector._call_chain)
14: extended_urllib.py:205(ExtendedUrllib.send_mutant)
15: urllib2.py:414(OpenerDirector._open)
16: web_spider.py:270(web_spider._should_output_extracted_url)
17: headers.py:34(Headers.__init__)
18: threadpool.py:78(Pool.map_multi_args)
19: threadpool.py:54(return_args.__call__)
Measurement #6 (w3af-10273-2014-02-09-02_34.cpu)
0: extended_urllib.py:434(ExtendedUrllib._send)
1: plugin.py:217(meth)
2: extended_urllib.py:252(ExtendedUrllib.GET)
3: urllib2.py:386(OpenerDirector.open)
4: threadpool.py:42(one_to_many.__call__)
5: web_spider.py:345(web_spider._verify_reference)
6: pool.py:64(mapstar)
7: web_spider.py:170(web_spider._urls_to_verify_generator)
8: url.py:173(URL.__init__)
9: base_consumer.py:42(crawl_infrastructure._wrapper)
10: pool.py:298(Pool.map_async)
11: HTTPResponse.py:127(from_httplib_resp)
12: url.py:47(URL.changed_wrapper)
13: threadpool.py:78(Pool.map_multi_args)
14: parser_cache.py:51(ParserCache.get_document_parser_for)
15: urllib2.py:374(OpenerDirector._call_chain)
16: web_spider.py:270(web_spider._should_output_extracted_url)
17: extended_urllib.py:205(ExtendedUrllib.send_mutant)
18: urllib2.py:414(OpenerDirector._open)
19: document_parser.py:40(DocumentParser.__init__)
Core status summary
Measurement #0 (w3af-10273-2014-02-09-02_22.core)
Audit queue input speed: 122.06
Requests per minute: 362
Audit queue output speed: 56.57
Crawl queue size: 0
Crawl queue input speed: 122.06
Crawl queue output speed: 148.23
Audit queue size: 81
Requests sent: 726
Measurement #1 (w3af-10273-2014-02-09-02_24.core)
Audit queue input speed: 114.33
Requests per minute: 402
Audit queue output speed: 48.3
Crawl queue size: 0
Crawl queue input speed: 114.33
Crawl queue output speed: 194.3
Audit queue size: 143
Requests sent: 1610
Measurement #2 (w3af-10273-2014-02-09-02_26.core)
Audit queue input speed: 61.14
Requests per minute: 361
Audit queue output speed: 55.62
Crawl queue size: 0
Crawl queue input speed: 61.14
Crawl queue output speed: 61.14
Audit queue size: 185
Requests sent: 2168
Measurement #3 (w3af-10273-2014-02-09-02_28.core)
Audit queue input speed: 68.11
Requests per minute: 335
Audit queue output speed: 45.21
Crawl queue size: 0
Crawl queue input speed: 68.11
Crawl queue output speed: 68.12
Audit queue size: 264
Requests sent: 2689
Measurement #4 (w3af-10273-2014-02-09-02_30.core)
Audit queue input speed: 66.02
Requests per minute: 321
Audit queue output speed: 56.65
Crawl queue size: 13
Crawl queue input speed: 66.02
Crawl queue output speed: 63.43
Audit queue size: 300
Requests sent: 3216
Measurement #5 (w3af-10273-2014-02-09-02_32.core)
Audit queue input speed: None
Requests per minute: 300
Audit queue output speed: None
Crawl queue size: 0
Crawl queue input speed: 47.39
Crawl queue output speed: 49.0
Audit queue size: None
Requests sent: 3613
Measurement #6 (w3af-10273-2014-02-09-02_34.core)
Audit queue input speed: None
Requests per minute: 284
Audit queue output speed: None
Crawl queue size: None
Crawl queue input speed: None
Crawl queue output speed: None
Audit queue size: None
Requests sent: 3981
Performance analysis
====================
Total memory size
0: 32.4MiB
1: 67.3MiB
2: 67.3MiB
Memory usage summary
First:
Total 201138 objects, 516 types, Total size = 32.4MiB (33995558 bytes)
Index Count % Size % Cum Max Kind
0 7500 3 8082336 23 23 196888 dict
1 61171 30 5746127 16 40 12467 str
2 1324 0 3303360 9 50 49488 module
3 37725 18 2647940 7 58 9236 unicode
4 18522 9 2370816 6 65 128 code
5 18943 9 2273160 6 71 120 function
6 2479 1 2241016 6 78 904 type
7 13952 6 1059872 3 81 3184 tuple
8 7509 3 854704 2 84 20192 list
9 524 0 468704 1 85 8424 set
10 1024 0 360448 1 86 352 U32
11 356 0 321824 0 87 904 Packet_metaclass
12 3423 1 301224 0 88 88 weakref
13 8685 4 208440 0 88 24 int
14 165 0 176736 0 89 1120 ByteEnumField
15 376 0 170648 0 89 12584 OrderedDict
16 122 0 133184 0 90 1120 BitField
17 110 0 122320 0 90 1112 Distribution
18 108 0 118272 0 91 1120 ShortEnumField
19 1428 0 114240 0 91 80 wrapper_descriptor
Before last:
Total 287156 objects, 570 types, Total size = 67.3MiB (70519089 bytes)
Index Count % Size % Cum Max Kind
0 60154 20 20661876 29 29 1624232 unicode
1 22196 7 12532448 17 47 196888 dict
2 70726 24 7423770 10 57 120508 str
3 5839 2 6473000 9 66 1112 URL
4 1325 0 3306768 4 71 49488 module
5 18532 6 2372096 3 74 128 code
6 18980 6 2277600 3 78 120 function
7 2484 0 2245536 3 81 904 type
8 18081 6 1841920 2 83 20192 list
9 5768 2 1708864 2 86 1064 QueryString
10 20345 7 1533344 2 88 3184 tuple
11 701 0 978248 1 89 8424 set
12 1024 0 360448 0 90 352 U32
13 356 0 321824 0 90 904 Packet_metaclass
14 3523 1 310024 0 91 88 weakref
15 619 0 305688 0 91 624 frame
16 272 0 292864 0 92 1112 FuzzableRequest
17 731 0 265288 0 92 1112 _Condition
18 9322 3 223728 0 92 24 int
19 1987 0 190752 0 93 96 lxml.etree._LogEntry
Operating System information
Load average: 5.9 6.79 4.25
Network
Bytes sent: 4.8M
Bytes received: 305.0M
Swap memory
% used (best: 0.0): 0.0
Pages per second (out) (best: 0): 0
Pages per second (in) (best: 0): 0
CPU usage by function
Measurement #0 (w3af-10272-2014-02-09-01_16.cpu)
0: cpu_usage.py:56(dump_data)
1: utils.py:34(dump_data_every_thread)
Measurement #1 (w3af-10272-2014-02-09-01_18.cpu)
0: extended_urllib.py:434(ExtendedUrllib._send)
1: extended_urllib.py:252(ExtendedUrllib.GET)
2: plugin.py:217(meth)
3: threadpool.py:42(one_to_many.__call__)
4: web_spider.py:345(web_spider._verify_reference)
5: urllib2.py:386(OpenerDirector.open)
6: pool.py:64(mapstar)
7: utils.py:34(dump_data_every_thread)
8: memory_usage.py:33(inner)
9: memory_usage.py:42(start_memory_profiling)
10: memory_usage.py:53(dump_objects)
11: scanner.py:123(dump_all_objects)
12: pstats.py:451(TupleComp)
13: base_consumer.py:42(crawl_infrastructure._wrapper)
14: url.py:173(URL.__init__)
15: urllib2.py:414(OpenerDirector._open)
16: HTTPResponse.py:127(from_httplib_resp)
17: urllib2.py:374(OpenerDirector._call_chain)
18: web_spider.py:170(web_spider._urls_to_verify_generator)
19: keepalive.py:779(HTTPHandler.http_open)
Measurement #2 (w3af-10272-2014-02-09-01_20.cpu)
0: extended_urllib.py:434(ExtendedUrllib._send)
1: extended_urllib.py:252(ExtendedUrllib.GET)
2: urllib2.py:386(OpenerDirector.open)
3: plugin.py:217(meth)
4: web_spider.py:345(web_spider._verify_reference)
5: threadpool.py:42(one_to_many.__call__)
6: pool.py:64(mapstar)
7: url.py:173(URL.__init__)
8: web_spider.py:170(web_spider._urls_to_verify_generator)
9: HTTPResponse.py:127(from_httplib_resp)
10: urllib2.py:414(OpenerDirector._open)
11: urllib2.py:374(OpenerDirector._call_chain)
12: base_consumer.py:42(crawl_infrastructure._wrapper)
13: utils.py:34(dump_data_every_thread)
14: url.py:47(URL.changed_wrapper)
15: memory_usage.py:33(inner)
16: memory_usage.py:42(start_memory_profiling)
17: memory_usage.py:53(dump_objects)
18: scanner.py:123(dump_all_objects)
19: pstats.py:451(TupleComp)
Measurement #3 (w3af-10272-2014-02-09-01_23.cpu)
0: extended_urllib.py:434(ExtendedUrllib._send)
1: plugin.py:217(meth)
2: extended_urllib.py:252(ExtendedUrllib.GET)
3: urllib2.py:386(OpenerDirector.open)
4: base_consumer.py:42(crawl_infrastructure._wrapper)
5: web_spider.py:345(web_spider._verify_reference)
6: url.py:173(URL.__init__)
7: threadpool.py:42(one_to_many.__call__)
8: web_spider.py:170(web_spider._urls_to_verify_generator)
9: pool.py:64(mapstar)
10: HTTPResponse.py:127(from_httplib_resp)
11: url.py:47(URL.changed_wrapper)
12: urllib2.py:414(OpenerDirector._open)
13: urllib2.py:374(OpenerDirector._call_chain)
14: web_spider.py:270(web_spider._should_output_extracted_url)
15: headers.py:34(Headers.__init__)
16: db.py:67(store_in_cache)
17: kv_container.py:45(QueryString.__init__)
18: url.py:306(URL.set_querystring)
19: parser_cache.py:51(ParserCache.get_document_parser_for)
Measurement #4 (w3af-10272-2014-02-09-01_25.cpu)
0: urllib2.py:386(OpenerDirector.open)
1: extended_urllib.py:434(ExtendedUrllib._send)
2: plugin.py:217(meth)
3: extended_urllib.py:252(ExtendedUrllib.GET)
4: url.py:173(URL.__init__)
5: web_spider.py:170(web_spider._urls_to_verify_generator)
6: base_consumer.py:42(crawl_infrastructure._wrapper)
7: url.py:47(URL.changed_wrapper)
8: web_spider.py:345(web_spider._verify_reference)
9: threadpool.py:42(one_to_many.__call__)
10: HTTPResponse.py:127(from_httplib_resp)
11: pool.py:64(mapstar)
12: web_spider.py:270(web_spider._should_output_extracted_url)
13: pool.py:298(Pool.map_async)
14: parser_cache.py:51(ParserCache.get_document_parser_for)
15: document_parser.py:40(DocumentParser.__init__)
16: html.py:45(HTMLParser.__init__)
17: sgml.py:62(HTMLParser.__init__)
18: sgml.py:137(HTMLParser._parse)
19: headers.py:34(Headers.__init__)
Core status summary
Measurement #0 (w3af-10272-2014-02-09-01_18.core)
Audit queue input speed: 139.36
Requests per minute: 510
Audit queue output speed: 64.43
Crawl queue size: 29
Crawl queue input speed: 139.37
Crawl queue output speed: 135.73
Audit queue size: 81
Requests sent: 1022
Measurement #1 (w3af-10272-2014-02-09-01_20.core)
Audit queue input speed: 137.82
Requests per minute: 412
Audit queue output speed: 47.49
Crawl queue size: 117
Crawl queue input speed: 137.82
Crawl queue output speed: 48.28
Audit queue size: 152
Requests sent: 1650
Measurement #2 (w3af-10272-2014-02-09-01_22.core)
Audit queue input speed: None
Requests per minute: 331
Audit queue output speed: None
Crawl queue size: None
Crawl queue input speed: None
Crawl queue output speed: None
Audit queue size: None
Requests sent: 1990
Measurement #3 (w3af-10272-2014-02-09-01_24.core)
Audit queue input speed: None
Requests per minute: 312
Audit queue output speed: None
Crawl queue size: None
Crawl queue input speed: None
Crawl queue output speed: None
Audit queue size: None
Requests sent: 2507
Performance analysis
====================
Total memory size
0: 32.4MiB
1: 75.2MiB
2: 75.2MiB
Memory usage summary
First:
Total 201092 objects, 516 types, Total size = 32.4MiB (33992123 bytes)
Index Count % Size % Cum Max Kind
0 7500 3 8082336 23 23 196888 dict
1 61154 30 5745388 16 40 12467 str
2 1324 0 3303360 9 50 49488 module
3 37725 18 2647940 7 58 9236 unicode
4 18522 9 2370816 6 65 128 code
5 18943 9 2273160 6 71 120 function
6 2479 1 2241016 6 78 904 type
7 13924 6 1057200 3 81 3184 tuple
8 7509 3 854704 2 84 20192 list
9 524 0 468704 1 85 8424 set
10 1024 0 360448 1 86 352 U32
11 356 0 321824 0 87 904 Packet_metaclass
12 3423 1 301224 0 88 88 weakref
13 8685 4 208440 0 88 24 int
14 165 0 176736 0 89 1120 ByteEnumField
15 376 0 170648 0 89 12584 OrderedDict
16 122 0 133184 0 90 1120 BitField
17 110 0 122320 0 90 1112 Distribution
18 108 0 118272 0 91 1120 ShortEnumField
19 1428 0 114240 0 91 80 wrapper_descriptor
Before last:
Total 292330 objects, 567 types, Total size = 75.2MiB (78810120 bytes)
Index Count % Size % Cum Max Kind
0 61531 21 27484324 34 34 1375644 unicode
1 23218 7 12834736 16 51 196888 dict
2 70993 24 7833113 9 61 343954 str
3 5773 1 6409208 8 69 1112 URL
4 1325 0 3306768 4 73 49488 module
5 18530 6 2371840 3 76 128 code
6 18989 6 2278680 2 79 120 function
7 2484 0 2245536 2 82 904 type
8 18974 6 1892504 2 84 20192 list
9 5724 1 1695072 2 86 1064 QueryString
10 21068 7 1587160 2 88 3184 tuple
11 689 0 935528 1 89 8424 set
12 565 0 620600 0 90 1112 FuzzableRequest
13 1024 0 360448 0 91 352 U32
14 724 0 356656 0 91 624 frame
15 356 0 321824 0 92 904 Packet_metaclass
16 3522 1 309936 0 92 88 weakref
17 628 0 274208 0 92 1064 Headers
18 643 0 231176 0 93 1112 _Condition
19 9262 3 222288 0 93 24 int
Operating System information
Load average: 8.49 8.48 5.02
Network
Bytes sent: 5.1M
Bytes received: 306.6M
Swap memory
% used (best: 0.0): 0.0
Pages per second (out) (best: 0): 0
Pages per second (in) (best: 0): 0
CPU usage by function
Measurement #0 (w3af-10284-2014-02-09-01_16.cpu)
0: cpu_usage.py:56(dump_data)
1: utils.py:34(dump_data_every_thread)
Measurement #1 (w3af-10284-2014-02-09-01_19.cpu)
0: extended_urllib.py:434(ExtendedUrllib._send)
1: extended_urllib.py:252(ExtendedUrllib.GET)
2: plugin.py:217(meth)
3: threadpool.py:42(one_to_many.__call__)
4: web_spider.py:345(web_spider._verify_reference)
5: pool.py:64(mapstar)
6: urllib2.py:386(OpenerDirector.open)
7: utils.py:34(dump_data_every_thread)
8: memory_usage.py:33(inner)
9: memory_usage.py:42(start_memory_profiling)
10: memory_usage.py:53(dump_objects)
11: scanner.py:123(dump_all_objects)
12: pstats.py:451(TupleComp)
13: HTTPResponse.py:127(from_httplib_resp)
14: url.py:173(URL.__init__)
15: urllib2.py:414(OpenerDirector._open)
16: urllib2.py:374(OpenerDirector._call_chain)
17: base_consumer.py:42(crawl_infrastructure._wrapper)
18: headers.py:34(Headers.__init__)
19: url.py:47(URL.changed_wrapper)
Measurement #2 (w3af-10284-2014-02-09-01_21.cpu)
0: web_spider.py:345(web_spider._verify_reference)
1: extended_urllib.py:434(ExtendedUrllib._send)
2: extended_urllib.py:252(ExtendedUrllib.GET)
3: plugin.py:217(meth)
4: threadpool.py:42(one_to_many.__call__)
5: urllib2.py:386(OpenerDirector.open)
6: pool.py:64(mapstar)
7: HTTPResponse.py:127(from_httplib_resp)
8: url.py:173(URL.__init__)
9: base_consumer.py:42(crawl_infrastructure._wrapper)
10: urllib2.py:374(OpenerDirector._call_chain)
11: urllib2.py:414(OpenerDirector._open)
12: headers.py:34(Headers.__init__)
13: web_spider.py:170(web_spider._urls_to_verify_generator)
14: url.py:47(URL.changed_wrapper)
15: cache.py:78(CacheHandler.http_response)
16: db.py:67(store_in_cache)
17: utils.py:34(dump_data_every_thread)
18: pool.py:298(Pool.map_async)
19: memory_usage.py:33(inner)
Measurement #3 (w3af-10284-2014-02-09-01_23.cpu)
0: extended_urllib.py:434(ExtendedUrllib._send)
1: web_spider.py:345(web_spider._verify_reference)
2: plugin.py:217(meth)
3: extended_urllib.py:252(ExtendedUrllib.GET)
4: threadpool.py:42(one_to_many.__call__)
5: urllib2.py:386(OpenerDirector.open)
6: pool.py:64(mapstar)
7: HTTPResponse.py:127(from_httplib_resp)
8: base_consumer.py:42(crawl_infrastructure._wrapper)
9: url.py:173(URL.__init__)
10: web_spider.py:170(web_spider._urls_to_verify_generator)
11: urllib2.py:374(OpenerDirector._call_chain)
12: urllib2.py:414(OpenerDirector._open)
13: headers.py:34(Headers.__init__)
14: url.py:47(URL.changed_wrapper)
15: threadpool.py:54(return_args.__call__)
16: cache.py:78(CacheHandler.http_response)
17: db.py:67(store_in_cache)
18: extended_urllib.py:205(ExtendedUrllib.send_mutant)
19: web_spider.py:270(web_spider._should_output_extracted_url)
Measurement #4 (w3af-10284-2014-02-09-01_25.cpu)
0: extended_urllib.py:434(ExtendedUrllib._send)
1: web_spider.py:345(web_spider._verify_reference)
2: plugin.py:217(meth)
3: extended_urllib.py:252(ExtendedUrllib.GET)
4: threadpool.py:42(one_to_many.__call__)
5: pool.py:64(mapstar)
6: urllib2.py:386(OpenerDirector.open)
7: web_spider.py:170(web_spider._urls_to_verify_generator)
8: base_consumer.py:42(crawl_infrastructure._wrapper)
9: url.py:173(URL.__init__)
10: HTTPResponse.py:127(from_httplib_resp)
11: threadpool.py:54(return_args.__call__)
12: threadpool.py:78(Pool.map_multi_args)
13: pool.py:298(Pool.map_async)
14: extended_urllib.py:205(ExtendedUrllib.send_mutant)
15: url.py:47(URL.changed_wrapper)
16: urllib2.py:374(OpenerDirector._call_chain)
17: urllib2.py:414(OpenerDirector._open)
18: web_spider.py:270(web_spider._should_output_extracted_url)
19: utils.py:34(dump_data_every_thread)
Core status summary
Measurement #0 (w3af-10284-2014-02-09-01_19.core)
Audit queue input speed: 130.39
Requests per minute: 445
Audit queue output speed: 49.05
Crawl queue size: 45
Crawl queue input speed: 130.39
Crawl queue output speed: 180.62
Audit queue size: 95
Requests sent: 892
Measurement #1 (w3af-10284-2014-02-09-01_21.core)
Audit queue input speed: 119.95
Requests per minute: 399
Audit queue output speed: 46.48
Crawl queue size: 0
Crawl queue input speed: 119.95
Crawl queue output speed: 214.42
Audit queue size: 180
Requests sent: 1601
Measurement #2 (w3af-10284-2014-02-09-01_23.core)
Audit queue input speed: 80.02
Requests per minute: 361
Audit queue output speed: 50.5
Crawl queue size: 0
Crawl queue input speed: 80.02
Crawl queue output speed: 116.76
Audit queue size: 252
Requests sent: 2169
Measurement #3 (w3af-10284-2014-02-09-01_25.core)
Audit queue input speed: 66.22
Requests per minute: 349
Audit queue output speed: 62.8
Crawl queue size: 1
Crawl queue input speed: 66.22
Crawl queue output speed: 66.57
Audit queue size: 262
Requests sent: 2802