1 <?xml version="1.0"?> <!-- -*- sgml -*- -->
2 <!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
3 "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
4 [ <!ENTITY % vg-entities SYSTEM "../../docs/xml/vg-entities.xml"> %vg-entities; ]>
7 <chapter id="cg-manual" xreflabel="Cachegrind: a cache and branch-prediction profiler">
8 <title>Cachegrind: a cache and branch-prediction profiler</title>
10 <para>To use this tool, you must specify
11 <option>--tool=cachegrind</option> on the
12 Valgrind command line.</para>
14 <sect1 id="cg-manual.overview" xreflabel="Overview">
15 <title>Overview</title>
17 <para>Cachegrind simulates how your program interacts with a machine's cache
18 hierarchy and (optionally) branch predictor. It simulates a machine with
19 independent first level instruction and data caches (I1 and D1), backed by a
20 unified second level cache (L2). This configuration is used by almost all
21 modern machines.</para>
24 It gathers the following statistics (abbreviations used for each statistic
25 is given in parentheses):</para>
28 <para>I cache reads (<computeroutput>Ir</computeroutput>,
29 which equals the number of instructions executed),
30 I1 cache read misses (<computeroutput>I1mr</computeroutput>) and
31 L2 cache instruction read misses (<computeroutput>I1mr</computeroutput>).
35 <para>D cache reads (<computeroutput>Dr</computeroutput>, which
36 equals the number of memory reads),
37 D1 cache read misses (<computeroutput>D1mr</computeroutput>), and
38 L2 cache data read misses (<computeroutput>D2mr</computeroutput>).
42 <para>D cache writes (<computeroutput>Dw</computeroutput>, which equals
43 the number of memory writes),
44 D1 cache write misses (<computeroutput>D1mw</computeroutput>), and
45 L2 cache data write misses (<computeroutput>D2mw</computeroutput>).
49 <para>Conditional branches executed (<computeroutput>Bc</computeroutput>) and
50 conditional branches mispredicted (<computeroutput>Bcm</computeroutput>).
54 <para>Indirect branches executed (<computeroutput>Bi</computeroutput>) and
55 indirect branches mispredicted (<computeroutput>Bim</computeroutput>).
60 <para>Note that D1 total accesses is given by
61 <computeroutput>D1mr</computeroutput> +
62 <computeroutput>D1mw</computeroutput>, and that L2 total
63 accesses is given by <computeroutput>I2mr</computeroutput> +
64 <computeroutput>D2mr</computeroutput> +
65 <computeroutput>D2mw</computeroutput>.
68 <para>These statistics are presented for the entire program and for each
69 function in the program. You can also annotate each line of source code in
70 the program with the counts that were caused directly by it.</para>
72 <para>On a modern machine, an L1 miss will typically cost
73 around 10 cycles, an L2 miss can cost as much as 200
74 cycles, and a mispredicted branch costs in the region of 10
75 to 30 cycles. Detailed cache and branch profiling can be very useful
76 for understanding how your program interacts with the machine and thus how
77 to make it faster.</para>
79 <para>Also, since one instruction cache read is performed per
80 instruction executed, you can find out how many instructions are
81 executed per line, which can be useful for traditional profiling.</para>
87 <sect1 id="cg-manual.profile"
88 xreflabel="Using Cachegrind, cg_annotate and cg_merge">
89 <title>Using Cachegrind, cg_annotate and cg_merge</title>
91 <para>First off, as for normal Valgrind use, you probably want to
92 compile with debugging info (the
93 <option>-g</option> option). But by contrast with
94 normal Valgrind use, you probably do want to turn
95 optimisation on, since you should profile your program as it will
96 be normally run.</para>
98 <para>Then, you need to run Cachegrind itself to gather the profiling
99 information, and then run cg_annotate to get a detailed presentation of that
100 information. As an optional intermediate step, you can use cg_merge to sum
101 together the outputs of multiple Cachegrind runs, into a single file which
102 you then use as the input for cg_annotate.</para>
105 <sect2 id="cg-manual.running-cachegrind" xreflabel="Running Cachegrind">
106 <title>Running Cachegrind</title>
108 <para>To run Cachegrind on a program <filename>prog</filename>, run:</para>
110 valgrind --tool=cachegrind prog
113 <para>The program will execute (slowly). Upon completion,
114 summary statistics that look like this will be printed:</para>
116 <programlisting><![CDATA[
117 ==31751== I refs: 27,742,716
118 ==31751== I1 misses: 276
119 ==31751== L2i misses: 275
120 ==31751== I1 miss rate: 0.0%
121 ==31751== L2i miss rate: 0.0%
123 ==31751== D refs: 15,430,290 (10,955,517 rd + 4,474,773 wr)
124 ==31751== D1 misses: 41,185 ( 21,905 rd + 19,280 wr)
125 ==31751== L2d misses: 23,085 ( 3,987 rd + 19,098 wr)
126 ==31751== D1 miss rate: 0.2% ( 0.1% + 0.4%)
127 ==31751== L2d miss rate: 0.1% ( 0.0% + 0.4%)
129 ==31751== L2 misses: 23,360 ( 4,262 rd + 19,098 wr)
130 ==31751== L2 miss rate: 0.0% ( 0.0% + 0.4%)]]></programlisting>
132 <para>Cache accesses for instruction fetches are summarised
133 first, giving the number of fetches made (this is the number of
134 instructions executed, which can be useful to know in its own
135 right), the number of I1 misses, and the number of L2 instruction
136 (<computeroutput>L2i</computeroutput>) misses.</para>
138 <para>Cache accesses for data follow. The information is similar
139 to that of the instruction fetches, except that the values are
140 also shown split between reads and writes (note each row's
141 <computeroutput>rd</computeroutput> and
142 <computeroutput>wr</computeroutput> values add up to the row's
145 <para>Combined instruction and data figures for the L2 cache
146 follow that. Note that the L2 miss rate is computed relative to the total
147 number of memory accesses, not the number of L1 misses. I.e. it is
148 <computeroutput>(I2mr + D2mr + D2mw) / (Ir + Dr + Dw)</computeroutput>
150 <computeroutput>(I2mr + D2mr + D2mw) / (I1mr + D1mr + D1mw)</computeroutput>
153 <para>Branch prediction statistics are not collected by default.
154 To do so, add the option <option>--branch-sim=yes</option>.</para>
159 <sect2 id="cg-manual.outputfile" xreflabel="Output File">
160 <title>Output File</title>
162 <para>As well as printing summary information, Cachegrind also writes
163 more detailed profiling information to a file. By default this file is named
164 <filename>cachegrind.out.<pid></filename> (where
165 <filename><pid></filename> is the program's process ID), but its name
166 can be changed with the <option>--cachegrind-out-file</option> option. This
167 file is human-readable, but is intended to be interpreted by the
168 accompanying program cg_annotate, described in the next section.</para>
170 <para>The default <computeroutput>.<pid></computeroutput> suffix
171 on the output file name serves two purposes. Firstly, it means you
172 don't have to rename old log files that you don't want to overwrite.
173 Secondly, and more importantly, it allows correct profiling with the
174 <option>--trace-children=yes</option> option of
175 programs that spawn child processes.</para>
177 <para>The output file can be big, many megabytes for large applications
178 built with full debugging information.</para>
184 <sect2 id="cg-manual.running-cg_annotate" xreflabel="Running cg_annotate">
185 <title>Running cg_annotate</title>
187 <para>Before using cg_annotate,
188 it is worth widening your window to be at least 120-characters
189 wide if possible, as the output lines can be quite long.</para>
191 <para>To get a function-by-function summary, run:</para>
193 <screen>cg_annotate <filename></screen>
195 <para>on a Cachegrind output file.</para>
200 <sect2 id="cg-manual.the-output-preamble" xreflabel="The Output Preamble">
201 <title>The Output Preamble</title>
203 <para>The first part of the output looks like this:</para>
205 <programlisting><![CDATA[
206 --------------------------------------------------------------------------------
207 I1 cache: 65536 B, 64 B, 2-way associative
208 D1 cache: 65536 B, 64 B, 2-way associative
209 L2 cache: 262144 B, 64 B, 8-way associative
210 Command: concord vg_to_ucode.c
211 Events recorded: Ir I1mr I2mr Dr D1mr D2mr Dw D1mw D2mw
212 Events shown: Ir I1mr I2mr Dr D1mr D2mr Dw D1mw D2mw
213 Event sort order: Ir I1mr I2mr Dr D1mr D2mr Dw D1mw D2mw
215 Chosen for annotation:
220 <para>This is a summary of the annotation options:</para>
225 <para>I1 cache, D1 cache, L2 cache: cache configuration. So
226 you know the configuration with which these results were
231 <para>Command: the command line invocation of the program
232 under examination.</para>
236 <para>Events recorded: which events were recorded.</para>
241 <para>Events shown: the events shown, which is a subset of the events
242 gathered. This can be adjusted with the
243 <option>--show</option> option.</para>
247 <para>Event sort order: the sort order in which functions are
248 shown. For example, in this case the functions are sorted
249 from highest <computeroutput>Ir</computeroutput> counts to
250 lowest. If two functions have identical
251 <computeroutput>Ir</computeroutput> counts, they will then be
252 sorted by <computeroutput>I1mr</computeroutput> counts, and
253 so on. This order can be adjusted with the
254 <option>--sort</option> option.</para>
256 <para>Note that this dictates the order the functions appear.
257 It is <emphasis>not</emphasis> the order in which the columns
258 appear; that is dictated by the "events shown" line (and can
259 be changed with the <option>--show</option>
264 <para>Threshold: cg_annotate
265 by default omits functions that cause very low counts
266 to avoid drowning you in information. In this case,
267 cg_annotate shows summaries the functions that account for
268 99% of the <computeroutput>Ir</computeroutput> counts;
269 <computeroutput>Ir</computeroutput> is chosen as the
270 threshold event since it is the primary sort event. The
271 threshold can be adjusted with the
272 <option>--threshold</option>
277 <para>Chosen for annotation: names of files specified
278 manually for annotation; in this case none.</para>
282 <para>Auto-annotation: whether auto-annotation was requested
283 via the <option>--auto=yes</option>
284 option. In this case no.</para>
292 <sect2 id="cg-manual.the-global"
293 xreflabel="The Global and Function-level Counts">
294 <title>The Global and Function-level Counts</title>
296 <para>Then follows summary statistics for the whole
299 <programlisting><![CDATA[
300 --------------------------------------------------------------------------------
301 Ir I1mr I2mr Dr D1mr D2mr Dw D1mw D2mw
302 --------------------------------------------------------------------------------
303 27,742,716 276 275 10,955,517 21,905 3,987 4,474,773 19,280 19,098 PROGRAM TOTALS]]></programlisting>
306 These are similar to the summary provided when Cachegrind finishes running.
309 <para>Then comes function-by-function statistics:</para>
311 <programlisting><![CDATA[
312 --------------------------------------------------------------------------------
313 Ir I1mr I2mr Dr D1mr D2mr Dw D1mw D2mw file:function
314 --------------------------------------------------------------------------------
315 8,821,482 5 5 2,242,702 1,621 73 1,794,230 0 0 getc.c:_IO_getc
316 5,222,023 4 4 2,276,334 16 12 875,959 1 1 concord.c:get_word
317 2,649,248 2 2 1,344,810 7,326 1,385 . . . vg_main.c:strcmp
318 2,521,927 2 2 591,215 0 0 179,398 0 0 concord.c:hash
319 2,242,740 2 2 1,046,612 568 22 448,548 0 0 ctype.c:tolower
320 1,496,937 4 4 630,874 9,000 1,400 279,388 0 0 concord.c:insert
321 897,991 51 51 897,831 95 30 62 1 1 ???:???
322 598,068 1 1 299,034 0 0 149,517 0 0 ../sysdeps/generic/lockfile.c:__flockfile
323 598,068 0 0 299,034 0 0 149,517 0 0 ../sysdeps/generic/lockfile.c:__funlockfile
324 598,024 4 4 213,580 35 16 149,506 0 0 vg_clientmalloc.c:malloc
325 446,587 1 1 215,973 2,167 430 129,948 14,057 13,957 concord.c:add_existing
326 341,760 2 2 128,160 0 0 128,160 0 0 vg_clientmalloc.c:vg_trap_here_WRAPPER
327 320,782 4 4 150,711 276 0 56,027 53 53 concord.c:init_hash_table
328 298,998 1 1 106,785 0 0 64,071 1 1 concord.c:create
329 149,518 0 0 149,516 0 0 1 0 0 ???:tolower@@GLIBC_2.0
330 149,518 0 0 149,516 0 0 1 0 0 ???:fgetc@@GLIBC_2.0
331 95,983 4 4 38,031 0 0 34,409 3,152 3,150 concord.c:new_word_node
332 85,440 0 0 42,720 0 0 21,360 0 0 vg_clientmalloc.c:vg_bogus_epilogue]]></programlisting>
336 <computeroutput>file_name:function_name</computeroutput> pair. If
337 a column contains only a dot it means the function never performs
338 that event (e.g. the third row shows that
339 <computeroutput>strcmp()</computeroutput> contains no
340 instructions that write to memory). The name
341 <computeroutput>???</computeroutput> is used if the the file name
342 and/or function name could not be determined from debugging
343 information. If most of the entries have the form
344 <computeroutput>???:???</computeroutput> the program probably
345 wasn't compiled with <option>-g</option>.</para>
347 <para>It is worth noting that functions will come both from
348 the profiled program (e.g. <filename>concord.c</filename>)
349 and from libraries (e.g. <filename>getc.c</filename>)</para>
354 <sect2 id="cg-manual.line-by-line" xreflabel="Line-by-line Counts">
355 <title>Line-by-line Counts</title>
357 <para>There are two ways to annotate source files -- by specifying them
358 manually as arguments to cg_annotate, or with the
359 <option>--auto=yes</option> option. For example, the output from running
360 <filename>cg_annotate <filename> concord.c</filename> for our example
361 produces the same output as above followed by an annotated version of
362 <filename>concord.c</filename>, a section of which looks like:</para>
364 <programlisting><![CDATA[
365 --------------------------------------------------------------------------------
366 -- User-annotated source: concord.c
367 --------------------------------------------------------------------------------
368 Ir I1mr I2mr Dr D1mr D2mr Dw D1mw D2mw
370 . . . . . . . . . void init_hash_table(char *file_name, Word_Node *table[])
372 . . . . . . . . . FILE *file_ptr;
373 . . . . . . . . . Word_Info *data;
374 1 0 0 . . . 1 1 1 int line = 1, i;
376 5 0 0 . . . 3 0 0 data = (Word_Info *) create(sizeof(Word_Info));
378 4,991 0 0 1,995 0 0 998 0 0 for (i = 0; i < TABLE_SIZE; i++)
379 3,988 1 1 1,994 0 0 997 53 52 table[i] = NULL;
381 . . . . . . . . . /* Open file, check it. */
382 6 0 0 1 0 0 4 0 0 file_ptr = fopen(file_name, "r");
383 2 0 0 1 0 0 . . . if (!(file_ptr)) {
384 . . . . . . . . . fprintf(stderr, "Couldn't open '%s'.\n", file_name);
385 1 1 1 . . . . . . exit(EXIT_FAILURE);
388 165,062 1 1 73,360 0 0 91,700 0 0 while ((line = get_word(data, line, file_ptr)) != EOF)
389 146,712 0 0 73,356 0 0 73,356 0 0 insert(data->;word, data->line, table);
391 4 0 0 1 0 0 2 0 0 free(data);
392 4 0 0 1 0 0 2 0 0 fclose(file_ptr);
393 3 0 0 2 0 0 . . . }]]></programlisting>
395 <para>(Although column widths are automatically minimised, a wide
396 terminal is clearly useful.)</para>
398 <para>Each source file is clearly marked
399 (<computeroutput>User-annotated source</computeroutput>) as
400 having been chosen manually for annotation. If the file was
401 found in one of the directories specified with the
402 <option>-I</option>/<option>--include</option> option, the directory
403 and file are both given.</para>
405 <para>Each line is annotated with its event counts. Events not
406 applicable for a line are represented by a dot. This is useful
407 for distinguishing between an event which cannot happen, and one
408 which can but did not.</para>
410 <para>Sometimes only a small section of a source file is
411 executed. To minimise uninteresting output, Cachegrind only shows
412 annotated lines and lines within a small distance of annotated
413 lines. Gaps are marked with the line numbers so you know which
414 part of a file the shown code comes from, eg:</para>
416 <programlisting><![CDATA[
417 (figures and code for line 704)
418 -- line 704 ----------------------------------------
419 -- line 878 ----------------------------------------
420 (figures and code for line 878)]]></programlisting>
422 <para>The amount of context to show around annotated lines is
423 controlled by the <option>--context</option>
426 <para>To get automatic annotation, use the <option>--auto=yes</option> option.
427 cg_annotate will automatically annotate every source file it can
428 find that is mentioned in the function-by-function summary.
429 Therefore, the files chosen for auto-annotation are affected by
430 the <option>--sort</option> and
431 <option>--threshold</option> options. Each
432 source file is clearly marked (<computeroutput>Auto-annotated
433 source</computeroutput>) as being chosen automatically. Any
434 files that could not be found are mentioned at the end of the
437 <programlisting><![CDATA[
438 ------------------------------------------------------------------
439 The following files chosen for auto-annotation could not be found:
440 ------------------------------------------------------------------
443 ../sysdeps/generic/lockfile.c]]></programlisting>
445 <para>This is quite common for library files, since libraries are
446 usually compiled with debugging information, but the source files
447 are often not present on a system. If a file is chosen for
448 annotation both manually and automatically, it
449 is marked as <computeroutput>User-annotated
450 source</computeroutput>. Use the
451 <option>-I</option>/<option>--include</option> option to tell Valgrind where
452 to look for source files if the filenames found from the debugging
453 information aren't specific enough.</para>
455 <para>Beware that cg_annotate can take some time to digest large
456 <filename>cachegrind.out.<pid></filename> files,
457 e.g. 30 seconds or more. Also beware that auto-annotation can
458 produce a lot of output if your program is large!</para>
463 <sect2 id="cg-manual.assembler" xreflabel="Annotating Assembly Code Programs">
464 <title>Annotating Assembly Code Programs</title>
466 <para>Valgrind can annotate assembly code programs too, or annotate
467 the assembly code generated for your C program. Sometimes this is
468 useful for understanding what is really happening when an
469 interesting line of C code is translated into multiple
472 <para>To do this, you just need to assemble your
473 <computeroutput>.s</computeroutput> files with assembly-level debug
474 information. You can use compile with the <option>-S</option> to compile C/C++
475 programs to assembly code, and then assemble the assembly code files with
476 <option>-g</option> to achieve this. You can then profile and annotate the
477 assembly code source files in the same way as C/C++ source files.</para>
481 <sect2 id="ms-manual.forkingprograms" xreflabel="Forking Programs">
482 <title>Forking Programs</title>
483 <para>If your program forks, the child will inherit all the profiling data that
484 has been gathered for the parent.</para>
486 <para>If the output file format string (controlled by
487 <option>--cachegrind-out-file</option>) does not contain <option>%p</option>,
488 then the outputs from the parent and child will be intermingled in a single
489 output file, which will almost certainly make it unreadable by
494 <sect2 id="cg-manual.annopts.warnings" xreflabel="cg_annotate Warnings">
495 <title>cg_annotate Warnings</title>
497 <para>There are a couple of situations in which
498 cg_annotate issues warnings.</para>
502 <para>If a source file is more recent than the
503 <filename>cachegrind.out.<pid></filename> file.
504 This is because the information in
505 <filename>cachegrind.out.<pid></filename> is only
506 recorded with line numbers, so if the line numbers change at
507 all in the source (e.g. lines added, deleted, swapped), any
508 annotations will be incorrect.</para>
511 <para>If information is recorded about line numbers past the
512 end of a file. This can be caused by the above problem,
513 i.e. shortening the source file while using an old
514 <filename>cachegrind.out.<pid></filename> file. If
515 this happens, the figures for the bogus lines are printed
516 anyway (clearly marked as bogus) in case they are
525 <sect2 id="cg-manual.annopts.things-to-watch-out-for"
526 xreflabel="Unusual Annotation Cases">
527 <title>Unusual Annotation Cases</title>
529 <para>Some odd things that can occur during annotation:</para>
533 <para>If annotating at the assembler level, you might see
534 something like this:</para>
535 <programlisting><![CDATA[
536 1 0 0 . . . . . . leal -12(%ebp),%eax
537 1 0 0 . . . 1 0 0 movl %eax,84(%ebx)
538 2 0 0 0 0 0 1 0 0 movl $1,-20(%ebp)
539 . . . . . . . . . .align 4,0x90
540 1 0 0 . . . . . . movl $.LnrB,%eax
541 1 0 0 . . . 1 0 0 movl %eax,-16(%ebp)]]></programlisting>
543 <para>How can the third instruction be executed twice when
544 the others are executed only once? As it turns out, it
545 isn't. Here's a dump of the executable, using
546 <computeroutput>objdump -d</computeroutput>:</para>
547 <programlisting><![CDATA[
548 8048f25: 8d 45 f4 lea 0xfffffff4(%ebp),%eax
549 8048f28: 89 43 54 mov %eax,0x54(%ebx)
550 8048f2b: c7 45 ec 01 00 00 00 movl $0x1,0xffffffec(%ebp)
551 8048f32: 89 f6 mov %esi,%esi
552 8048f34: b8 08 8b 07 08 mov $0x8078b08,%eax
553 8048f39: 89 45 f0 mov %eax,0xfffffff0(%ebp)]]></programlisting>
555 <para>Notice the extra <computeroutput>mov
556 %esi,%esi</computeroutput> instruction. Where did this come
557 from? The GNU assembler inserted it to serve as the two
558 bytes of padding needed to align the <computeroutput>movl
559 $.LnrB,%eax</computeroutput> instruction on a four-byte
560 boundary, but pretended it didn't exist when adding debug
561 information. Thus when Valgrind reads the debug info it
562 thinks that the <computeroutput>movl
563 $0x1,0xffffffec(%ebp)</computeroutput> instruction covers the
564 address range 0x8048f2b--0x804833 by itself, and attributes
565 the counts for the <computeroutput>mov
566 %esi,%esi</computeroutput> to it.</para>
570 I think this isn't true any more, not since cost centres were moved from
571 being associated with instruction addresses to being associated with
574 <para>Inlined functions can cause strange results in the
575 function-by-function summary. If a function
576 <computeroutput>inline_me()</computeroutput> is defined in
577 <filename>foo.h</filename> and inlined in the functions
578 <computeroutput>f1()</computeroutput>,
579 <computeroutput>f2()</computeroutput> and
580 <computeroutput>f3()</computeroutput> in
581 <filename>bar.c</filename>, there will not be a
582 <computeroutput>foo.h:inline_me()</computeroutput> function
583 entry. Instead, there will be separate function entries for
584 each inlining site, i.e.
585 <computeroutput>foo.h:f1()</computeroutput>,
586 <computeroutput>foo.h:f2()</computeroutput> and
587 <computeroutput>foo.h:f3()</computeroutput>. To find the
589 <computeroutput>foo.h:inline_me()</computeroutput>, add up
590 the counts from each entry.</para>
592 <para>The reason for this is that although the debug info
593 output by GCC indicates the switch from
594 <filename>bar.c</filename> to <filename>foo.h</filename>, it
595 doesn't indicate the name of the function in
596 <filename>foo.h</filename>, so Valgrind keeps using the old
602 <para>Sometimes, the same filename might be represented with
603 a relative name and with an absolute name in different parts
604 of the debug info, eg:
605 <filename>/home/user/proj/proj.h</filename> and
606 <filename>../proj.h</filename>. In this case, if you use
607 auto-annotation, the file will be annotated twice with the
608 counts split between the two.</para>
612 <para>Files with more than 65,535 lines cause difficulties
613 for the Stabs-format debug info reader. This is because the line
614 number in the <computeroutput>struct nlist</computeroutput>
615 defined in <filename>a.out.h</filename> under Linux is only a
616 16-bit value. Valgrind can handle some files with more than
617 65,535 lines correctly by making some guesses to identify
618 line number overflows. But some cases are beyond it, in
619 which case you'll get a warning message explaining that
620 annotations for the file might be incorrect.</para>
622 <para>If you are using GCC 3.1 or later, this is most likely
623 irrelevant, since GCC switched to using the more modern DWARF2
624 format by default at version 3.1. DWARF2 does not have any such
625 limitations on line numbers.</para>
629 <para>If you compile some files with
630 <option>-g</option> and some without, some
631 events that take place in a file without debug info could be
632 attributed to the last line of a file with debug info
633 (whichever one gets placed before the non-debug-info file in
634 the executable).</para>
639 <para>This list looks long, but these cases should be fairly
645 <sect2 id="cg-manual.cg_merge" xreflabel="cg_merge">
646 <title>Merging Profiles with cg_merge</title>
649 cg_merge is a simple program which
650 reads multiple profile files, as created by Cachegrind, merges them
651 together, and writes the results into another file in the same format.
652 You can then examine the merged results using
653 <computeroutput>cg_annotate <filename></computeroutput>, as
654 described above. The merging functionality might be useful if you
655 want to aggregate costs over multiple runs of the same program, or
656 from a single parallel run with multiple instances of the same
660 cg_merge is invoked as follows:
663 <programlisting><![CDATA[
664 cg_merge -o outputfile file1 file2 file3 ...]]></programlisting>
667 It reads and checks <computeroutput>file1</computeroutput>, then read
668 and checks <computeroutput>file2</computeroutput> and merges it into
669 the running totals, then the same with
670 <computeroutput>file3</computeroutput>, etc. The final results are
671 written to <computeroutput>outputfile</computeroutput>, or to standard
672 out if no output file is specified.</para>
675 Costs are summed on a per-function, per-line and per-instruction
676 basis. Because of this, the order in which the input files does not
677 matter, although you should take care to only mention each file once,
678 since any file mentioned twice will be added in twice.</para>
681 cg_merge does not attempt to check
682 that the input files come from runs of the same executable. It will
683 happily merge together profile files from completely unrelated
684 programs. It does however check that the
685 <computeroutput>Events:</computeroutput> lines of all the inputs are
686 identical, so as to ensure that the addition of costs makes sense.
687 For example, it would be nonsensical for it to add a number indicating
688 D1 read references to a number from a different file indicating L2
692 A number of other syntax and sanity checks are done whilst reading the
693 inputs. cg_merge will stop and
694 attempt to print a helpful error message if any of the input files
695 fail these checks.</para>
704 <sect1 id="cg-manual.cgopts" xreflabel="Cachegrind Command-line Options">
705 <title>Cachegrind Command-line Options</title>
707 <!-- start of xi:include in the manpage -->
708 <para>Cachegrind-specific options are:</para>
710 <variablelist id="cg.opts.list">
712 <varlistentry id="opt.I1" xreflabel="--I1">
714 <option><![CDATA[--I1=<size>,<associativity>,<line size> ]]></option>
717 <para>Specify the size, associativity and line size of the level 1
718 instruction cache. </para>
722 <varlistentry id="opt.D1" xreflabel="--D1">
724 <option><![CDATA[--D1=<size>,<associativity>,<line size> ]]></option>
727 <para>Specify the size, associativity and line size of the level 1
732 <varlistentry id="opt.L2" xreflabel="--L2">
734 <option><![CDATA[--L2=<size>,<associativity>,<line size> ]]></option>
737 <para>Specify the size, associativity and line size of the level 2
742 <varlistentry id="opt.cache-sim" xreflabel="--cache-sim">
744 <option><![CDATA[--cache-sim=no|yes [yes] ]]></option>
747 <para>Enables or disables collection of cache access and miss
752 <varlistentry id="opt.branch-sim" xreflabel="--branch-sim">
754 <option><![CDATA[--branch-sim=no|yes [no] ]]></option>
757 <para>Enables or disables collection of branch instruction and
758 misprediction counts. By default this is disabled as it
759 slows Cachegrind down by approximately 25%. Note that you
760 cannot specify <option>--cache-sim=no</option>
761 and <option>--branch-sim=no</option>
762 together, as that would leave Cachegrind with no
763 information to collect.</para>
767 <varlistentry id="opt.cachegrind-out-file" xreflabel="--cachegrind-out-file">
769 <option><![CDATA[--cachegrind-out-file=<file> ]]></option>
772 <para>Write the profile data to
773 <computeroutput>file</computeroutput> rather than to the default
775 <filename>cachegrind.out.<pid></filename>. The
776 <option>%p</option> and <option>%q</option> format specifiers
777 can be used to embed the process ID and/or the contents of an
778 environment variable in the name, as is the case for the core
779 option <option><xref linkend="opt.log-file"/></option>.
785 <!-- end of xi:include in the manpage -->
791 <sect1 id="cg-manual.annopts" xreflabel="cg_annotate Command-line Options">
792 <title>cg_annotate Command-line Options</title>
794 <!-- start of xi:include in the manpage -->
795 <variablelist id="cg_annotate.opts.list">
799 <option><![CDATA[-h --help ]]></option>
802 <para>Show the help message.</para>
808 <option><![CDATA[--version ]]></option>
811 <para>Show the version number.</para>
817 <option><![CDATA[--show=A,B,C [default: all, using order in
818 cachegrind.out.<pid>] ]]></option>
821 <para>Specifies which events to show (and the column
822 order). Default is to use all present in the
823 <filename>cachegrind.out.<pid></filename> file (and
824 use the order in the file). Useful if you want to concentrate on, for
825 example, I cache misses (<option>--show=I1mr,I2mr</option>), or data
826 read misses (<option>--show=D1mr,D2mr</option>), or L2 data misses
827 (<option>--show=D2mr,D2mw</option>). Best used in conjunction with
828 <option>--sort</option>.</para>
834 <option><![CDATA[--sort=A,B,C [default: order in
835 cachegrind.out.<pid>] ]]></option>
838 <para>Specifies the events upon which the sorting of the
839 function-by-function entries will be based.</para>
845 <option><![CDATA[--threshold=X [default: 99%] ]]></option>
848 <para>Sets the threshold for the function-by-function
849 summary. Functions are shown that account for more than X%
850 of the primary sort event. If auto-annotating, also affects
851 which files are annotated.</para>
853 <para>Note: thresholds can be set for more than one of the
854 events by appending any events for the
855 <option>--sort</option> option with a colon
856 and a number (no spaces, though). E.g. if you want to see
857 the functions that cover 99% of L2 read misses and 99% of L2
858 write misses, use this option:</para>
859 <para><option>--sort=D2mr:99,D2mw:99</option></para>
865 <option><![CDATA[--auto=<no|yes> [default: no] ]]></option>
868 <para>When enabled, automatically annotates every file that
869 is mentioned in the function-by-function summary that can be
870 found. Also gives a list of those that couldn't be found.</para>
876 <option><![CDATA[--context=N [default: 8] ]]></option>
879 <para>Print N lines of context before and after each
880 annotated line. Avoids printing large sections of source
881 files that were not executed. Use a large number
882 (e.g. 100000) to show all source lines.</para>
888 <option><![CDATA[-I<dir> --include=<dir> [default: none] ]]></option>
891 <para>Adds a directory to the list in which to search for
892 files. Multiple <option>-I</option>/<option>--include</option>
893 options can be given to add multiple directories.</para>
898 <!-- end of xi:include in the manpage -->
904 <sect1 id="cg-manual.acting-on"
905 xreflabel="Acting on Cachegrind's Information">
906 <title>Acting on Cachegrind's Information</title>
908 Cachegrind gives you lots of information, but acting on that information
909 isn't always easy. Here are some rules of thumb that we have found to be
913 First of all, the global hit/miss counts and miss rates are not that useful.
914 If you have multiple programs or multiple runs of a program, comparing the
915 numbers might identify if any are outliers and worthy of closer
916 investigation. Otherwise, they're not enough to act on.</para>
919 The function-by-function counts are more useful to look at, as they pinpoint
920 which functions are causing large numbers of counts. However, beware that
921 inlining can make these counts misleading. If a function
922 <function>f</function> is always inlined, counts will be attributed to the
923 functions it is inlined into, rather than itself. However, if you look at
924 the line-by-line annotations for <function>f</function> you'll see the
925 counts that belong to <function>f</function>. (This is hard to avoid, it's
926 how the debug info is structured.) So it's worth looking for large numbers
927 in the line-by-line annotations.</para>
930 The line-by-line source code annotations are much more useful. In our
931 experience, the best place to start is by looking at the
932 <computeroutput>Ir</computeroutput> numbers. They simply measure how many
933 instructions were executed for each line, and don't include any cache
934 information, but they can still be very useful for identifying
938 After that, we have found that L2 misses are typically a much bigger source
939 of slow-downs than L1 misses. So it's worth looking for any snippets of
940 code with high <computeroutput>D2mr</computeroutput> or
941 <computeroutput>D2mw</computeroutput> counts. (You can use
943 --sort=D2mr</option> with cg_annotate to focus just on
944 <literal>D2mr</literal> counts, for example.) If you find any, it's still
945 not always easy to work out how to improve things. You need to have a
946 reasonable understanding of how caches work, the principles of locality, and
947 your program's data access patterns. Improving things may require
948 redesigning a data structure, for example.</para>
951 Looking at the <computeroutput>Bcm</computeroutput> and
952 <computeroutput>Bim</computeroutput> misses can also be helpful.
953 In particular, <computeroutput>Bim</computeroutput> misses are often caused
954 by <literal>switch</literal> statements, and in some cases these
955 <literal>switch</literal> statements can be replaced with table-driven code.
956 For example, you might replace code like this:</para>
958 <programlisting><![CDATA[
971 <para>with code like this:</para>
973 <programlisting><![CDATA[
976 enum E table[] = { 1, 2, 3 };
983 This is obviously a contrived example, but the basic principle applies in a
984 wide variety of situations.</para>
987 In short, Cachegrind can tell you where some of the bottlenecks in your code
988 are, but it can't tell you how to fix them. You have to work that out for
989 yourself. But at least you have the information!
995 <sect1 id="cg-manual.sim-details"
996 xreflabel="Simulation Details">
997 <title>Simulation Details</title>
999 This section talks about details you don't need to know about in order to
1000 use Cachegrind, but may be of interest to some people.
1003 <sect2 id="cache-sim" xreflabel="Cache Simulation Specifics">
1004 <title>Cache Simulation Specifics</title>
1006 <para>Specific characteristics of the cache simulation are as
1012 <para>Write-allocate: when a write miss occurs, the block
1013 written to is brought into the D1 cache. Most modern caches
1014 have this property.</para>
1018 <para>Bit-selection hash function: the set of line(s) in the cache
1019 to which a memory block maps is chosen by the middle bits
1020 M--(M+N-1) of the byte address, where:</para>
1023 <para>line size = 2^M bytes</para>
1026 <para>(cache size / line size / associativity) = 2^N bytes</para>
1032 <para>Inclusive L2 cache: the L2 cache typically replicates all
1033 the entries of the L1 caches, because fetching into L1 involves
1034 fetching into L2 first (this does not guarantee strict inclusiveness,
1035 as lines evicted from L2 still could reside in L1). This is
1036 standard on Pentium chips, but AMD Opterons, Athlons and Durons
1037 use an exclusive L2 cache that only holds
1038 blocks evicted from L1. Ditto most modern VIA CPUs.</para>
1043 <para>The cache configuration simulated (cache size,
1044 associativity and line size) is determined automatically using
1045 the x86 CPUID instruction. If you have a machine that (a)
1046 doesn't support the CPUID instruction, or (b) supports it in an
1047 early incarnation that doesn't give any cache information, then
1048 Cachegrind will fall back to using a default configuration (that
1049 of a model 3/4 Athlon). Cachegrind will tell you if this
1050 happens. You can manually specify one, two or all three levels
1051 (I1/D1/L2) of the cache from the command line using the
1052 <option>--I1</option>,
1053 <option>--D1</option> and
1054 <option>--L2</option> options.
1055 For cache parameters to be valid for simulation, the number
1056 of sets (with associativity being the number of cache lines in
1057 each set) has to be a power of two.</para>
1059 <para>On PowerPC platforms
1060 Cachegrind cannot automatically
1061 determine the cache configuration, so you will
1062 need to specify it with the
1063 <option>--I1</option>,
1064 <option>--D1</option> and
1065 <option>--L2</option> options.</para>
1068 <para>Other noteworthy behaviour:</para>
1072 <para>References that straddle two cache lines are treated as
1076 <para>If both blocks hit --> counted as one hit</para>
1079 <para>If one block hits, the other misses --> counted
1083 <para>If both blocks miss --> counted as one miss (not
1090 <para>Instructions that modify a memory location
1091 (e.g. <computeroutput>inc</computeroutput> and
1092 <computeroutput>dec</computeroutput>) are counted as doing
1093 just a read, i.e. a single data reference. This may seem
1094 strange, but since the write can never cause a miss (the read
1095 guarantees the block is in the cache) it's not very
1098 <para>Thus it measures not the number of times the data cache
1099 is accessed, but the number of times a data cache miss could
1105 <para>If you are interested in simulating a cache with different
1106 properties, it is not particularly hard to write your own cache
1107 simulator, or to modify the existing ones in
1108 <computeroutput>cg_sim.c</computeroutput>. We'd be
1109 interested to hear from anyone who does.</para>
1114 <sect2 id="branch-sim" xreflabel="Branch Simulation Specifics">
1115 <title>Branch Simulation Specifics</title>
1117 <para>Cachegrind simulates branch predictors intended to be
1118 typical of mainstream desktop/server processors of around 2004.</para>
1120 <para>Conditional branches are predicted using an array of 16384 2-bit
1121 saturating counters. The array index used for a branch instruction is
1122 computed partly from the low-order bits of the branch instruction's
1123 address and partly using the taken/not-taken behaviour of the last few
1124 conditional branches. As a result the predictions for any specific
1125 branch depend both on its own history and the behaviour of previous
1126 branches. This is a standard technique for improving prediction
1129 <para>For indirect branches (that is, jumps to unknown destinations)
1130 Cachegrind uses a simple branch target address predictor. Targets are
1131 predicted using an array of 512 entries indexed by the low order 9
1132 bits of the branch instruction's address. Each branch is predicted to
1133 jump to the same address it did last time. Any other behaviour causes
1134 a mispredict.</para>
1136 <para>More recent processors have better branch predictors, in
1137 particular better indirect branch predictors. Cachegrind's predictor
1138 design is deliberately conservative so as to be representative of the
1139 large installed base of processors which pre-date widespread
1140 deployment of more sophisticated indirect branch predictors. In
1141 particular, late model Pentium 4s (Prescott), Pentium M, Core and Core
1142 2 have more sophisticated indirect branch predictors than modelled by
1145 <para>Cachegrind does not simulate a return stack predictor. It
1146 assumes that processors perfectly predict function return addresses,
1147 an assumption which is probably close to being true.</para>
1149 <para>See Hennessy and Patterson's classic text "Computer
1150 Architecture: A Quantitative Approach", 4th edition (2007), Section
1151 2.3 (pages 80-89) for background on modern branch predictors.</para>
1155 <sect2 id="cg-manual.annopts.accuracy" xreflabel="Accuracy">
1156 <title>Accuracy</title>
1158 <para>Valgrind's cache profiling has a number of
1159 shortcomings:</para>
1163 <para>It doesn't account for kernel activity -- the effect of system
1164 calls on the cache and branch predictor contents is ignored.</para>
1168 <para>It doesn't account for other process activity.
1169 This is probably desirable when considering a single
1174 <para>It doesn't account for virtual-to-physical address
1175 mappings. Hence the simulation is not a true
1176 representation of what's happening in the
1177 cache. Most caches and branch predictors are physically indexed, but
1178 Cachegrind simulates caches using virtual addresses.</para>
1182 <para>It doesn't account for cache misses not visible at the
1183 instruction level, e.g. those arising from TLB misses, or
1184 speculative execution.</para>
1188 <para>Valgrind will schedule
1189 threads differently from how they would be when running natively.
1190 This could warp the results for threaded programs.</para>
1194 <para>The x86/amd64 instructions <computeroutput>bts</computeroutput>,
1195 <computeroutput>btr</computeroutput> and
1196 <computeroutput>btc</computeroutput> will incorrectly be
1197 counted as doing a data read if both the arguments are
1198 registers, eg:</para>
1199 <programlisting><![CDATA[
1200 btsl %eax, %edx]]></programlisting>
1202 <para>This should only happen rarely.</para>
1206 <para>x86/amd64 FPU instructions with data sizes of 28 and 108 bytes
1207 (e.g. <computeroutput>fsave</computeroutput>) are treated as
1208 though they only access 16 bytes. These instructions seem to
1209 be rare so hopefully this won't affect accuracy much.</para>
1214 <para>Another thing worth noting is that results are very sensitive.
1215 Changing the size of the the executable being profiled, or the sizes
1216 of any of the shared libraries it uses, or even the length of their
1217 file names, can perturb the results. Variations will be small, but
1218 don't expect perfectly repeatable results if your program changes at
1221 <para>More recent GNU/Linux distributions do address space
1222 randomisation, in which identical runs of the same program have their
1223 shared libraries loaded at different locations, as a security measure.
1224 This also perturbs the results.</para>
1226 <para>While these factors mean you shouldn't trust the results to
1227 be super-accurate, they should be close enough to be useful.</para>
1235 <sect1 id="cg-manual.impl-details"
1236 xreflabel="Implementation Details">
1237 <title>Implementation Details</title>
1239 This section talks about details you don't need to know about in order to
1240 use Cachegrind, but may be of interest to some people.
1243 <sect2 id="cg-manual.impl-details.how-cg-works"
1244 xreflabel="How Cachegrind Works">
1245 <title>How Cachegrind Works</title>
1246 <para>The best reference for understanding how Cachegrind works is chapter 3 of
1247 "Dynamic Binary Analysis and Instrumentation", by Nicholas Nethercote. It
1248 is available on the <ulink url="&vg-pubs-url;">Valgrind publications
1249 page</ulink>.</para>
1252 <sect2 id="cg-manual.impl-details.file-format"
1253 xreflabel="Cachegrind Output File Format">
1254 <title>Cachegrind Output File Format</title>
1255 <para>The file format is fairly straightforward, basically giving the
1256 cost centre for every line, grouped by files and
1257 functions. It's also totally generic and self-describing, in the sense that
1258 it can be used for any events that can be counted on a line-by-line basis,
1259 not just cache and branch predictor events. For example, earlier versions
1260 of Cachegrind didn't have a branch predictor simulation. When this was
1261 added, the file format didn't need to change at all. So the format (and
1262 consequently, cg_annotate) could be used by other tools.</para>
1264 <para>The file format:</para>
1265 <programlisting><![CDATA[
1266 file ::= desc_line* cmd_line events_line data_line+ summary_line
1267 desc_line ::= "desc:" ws? non_nl_string
1268 cmd_line ::= "cmd:" ws? cmd
1269 events_line ::= "events:" ws? (event ws)+
1270 data_line ::= file_line | fn_line | count_line
1271 file_line ::= "fl=" filename
1272 fn_line ::= "fn=" fn_name
1273 count_line ::= line_num ws? (count ws)+
1274 summary_line ::= "summary:" ws? (count ws)+
1275 count ::= num | "."]]></programlisting>
1280 <para><computeroutput>non_nl_string</computeroutput> is any
1281 string not containing a newline.</para>
1284 <para><computeroutput>cmd</computeroutput> is a string holding the
1285 command line of the profiled program.</para>
1288 <para><computeroutput>event</computeroutput> is a string containing
1289 no whitespace.</para>
1292 <para><computeroutput>filename</computeroutput> and
1293 <computeroutput>fn_name</computeroutput> are strings.</para>
1296 <para><computeroutput>num</computeroutput> and
1297 <computeroutput>line_num</computeroutput> are decimal
1301 <para><computeroutput>ws</computeroutput> is whitespace.</para>
1305 <para>The contents of the "desc:" lines are printed out at the top
1306 of the summary. This is a generic way of providing simulation
1307 specific information, e.g. for giving the cache configuration for
1308 cache simulation.</para>
1310 <para>More than one line of info can be presented for each file/fn/line number.
1311 In such cases, the counts for the named events will be accumulated.</para>
1313 <para>Counts can be "." to represent zero. This makes the files easier for
1314 humans to read.</para>
1316 <para>The number of counts in each
1317 <computeroutput>line</computeroutput> and the
1318 <computeroutput>summary_line</computeroutput> should not exceed
1319 the number of events in the
1320 <computeroutput>event_line</computeroutput>. If the number in
1321 each <computeroutput>line</computeroutput> is less, cg_annotate
1322 treats those missing as though they were a "." entry. This saves space.
1325 <para>A <computeroutput>file_line</computeroutput> changes the
1326 current file name. A <computeroutput>fn_line</computeroutput>
1327 changes the current function name. A
1328 <computeroutput>count_line</computeroutput> contains counts that
1329 pertain to the current filename/fn_name. A "fn="
1330 <computeroutput>file_line</computeroutput> and a
1331 <computeroutput>fn_line</computeroutput> must appear before any
1332 <computeroutput>count_line</computeroutput>s to give the context
1333 of the first <computeroutput>count_line</computeroutput>s.</para>
1335 <para>Each <computeroutput>file_line</computeroutput> will normally be
1336 immediately followed by a <computeroutput>fn_line</computeroutput>. But it
1337 doesn't have to be.</para>
1339 <para>The summary line is redundant, because it just holds the total counts
1340 for each event. But this serves as a useful sanity check of the data; if
1341 the totals for each event don't match the summary line, something has gone