Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.

Teams

Q&A for work

Connect and share knowledge within a single location that is structured and easy to search.

I am using Docker containers based on the "ubuntu" tag and cannot get linux perf tool to display debugging symbols.

Here is what I'm doing to demonstrate the problem.

First I start a container, here with an interactive shell.

$ docker run -t -i ubuntu:14.04 /bin/bash
Then from the container prompt I install linux perf tool.
$ apt-get update
$ apt-get install -y linux-tools-common linux-tools-generic linux-tools-`uname -r`
I can now use the perf tool. My kernel is 3.16.0-77-generic.
Now I'll install gcc, compile a test program, and try to run it under perf record.
$ apt-get install -y gcc
I paste in the test program into test.c:
#include <stdio.h>
int function(int i) {
    int j;
    for(j = 2; j <= i / 2; j++) {
        if (i % j == 0) {
            return 0;
    return 1;
int main() {
    int i;
    for(i = 2; i < 100000; i++) {
        if(function(i)) {
            printf("%d\n", i);
Then compile, run, and report:
$ gcc -g -O0 test.c && perf record ./a.out && perf report
The output looks something like this:
  72.38%  a.out  a.out              [.] 0x0000000000000544
   8.37%  a.out  a.out              [.] 0x000000000000055a
   8.30%  a.out  a.out              [.] 0x000000000000053d
   7.81%  a.out  a.out              [.] 0x0000000000000551
   0.40%  a.out  a.out              [.] 0x0000000000000540
This does not have symbols, even though the executable does have symbol information.
Doing the same general steps outside the container works fine, and shows something like this:
96.96%  a.out  a.out             [.] function 
0.35%  a.out  libc-2.19.so       [.] _IO_file_xsputn@@GLIBC_2.2.5
0.14%  a.out  [kernel.kallsyms]  [k] update_curr
0.12%  a.out  [kernel.kallsyms]  [k] update_cfs_shares
0.11%  a.out  [kernel.kallsyms]  [k] _raw_spin_lock_irqsave                 
In the host system I have already turned on kernel symbols by becoming root and doing:
$ echo 0 > /proc/sys/kernel/kptr_restrict 
How do I get the containerized version to work properly and show debugging symbols?
Running the container with -v /:/host flag and running perf report in the container with --symfs /host flag fixes it:
 96.59%  a.out  a.out              [.] function
  2.93%  a.out  [kernel.kallsyms]  [k] 0xffffffff8105144a
  0.13%  a.out  [nvidia]           [k] 0x00000000002eda57
  0.11%  a.out  libc-2.19.so       [.] vfprintf
  0.11%  a.out  libc-2.19.so       [.] 0x0000000000049980
  0.09%  a.out  a.out              [.] main
  0.02%  a.out  libc-2.19.so       [.] _IO_file_write
  0.02%  a.out  libc-2.19.so       [.] write
Part of the reason why it doesn't work as is? The output from perf script sort of sheds some light on this:
           a.out    24 3374818.880960: cycles:  ffffffff81141140 __perf_event__output_id_sample ([kernel.kallsyms])
           a.out    24 3374818.881012: cycles:  ffffffff817319fd _raw_spin_lock_irqsave ([kernel.kallsyms])
           a.out    24 3374818.882217: cycles:  ffffffff8109aba3 ttwu_do_activate.constprop.75 ([kernel.kallsyms])
           a.out    24 3374818.884071: cycles:            40053d [unknown] (/var/lib/docker/aufs/diff/9bd2d4389cf7ad185405245b1f5c7d24d461bd565757880bfb4f970d3f4f7915/a.out)
           a.out    24 3374818.885329: cycles:            400544 [unknown] (/var/lib/docker/aufs/diff/9bd2d4389cf7ad185405245b1f5c7d24d461bd565757880bfb4f970d3f4f7915/a.out)
Note the /var/lib/docker/aufs path.  That's from the host so it won't exist in the container and you need to help perf report to locate it.  This likely happens because the mmap events are tracked by perf outside of any cgroup and perf does not attempt to remap the paths.
Another option is to run perf host-side, like sudo perf record -a docker run -ti <container name>.  But the collection has to be system-wide here (the -a flag) as containers are spawned by docker daemon process which is not in the process hierarchy of the docker client tool we run here. 
                Thank you, that was exactly it. To add to the solution, in order to get kernel symbols you also need to add: --kallsyms=/proc/kallsyms.
– Nathan Whitehead
                Aug 15, 2016 at 17:43
                In case it helps anyone else: an even simpler way is to call docker run with -v /var/lib/docker/:/var/lib/docker, that makes it so perf does not require any special arguments to resolve symbols correctly.
– Nathan Whitehead
                Aug 15, 2016 at 23:14
Another way that doesn't require changing how you run the container (so you can profile an already running process) is to mount container's root on host using bindfs:
bindfs /proc/$(docker inspect --format {{.State.Pid}} $CONTAINER_ID)/root /foo
Then run perf report as perf report --symfs /foo
You'll have to run perf record system wide, but you can restrict it to only collect events for the specific container:
perf record -g -a -F 100 -e cpu-clock -G docker/$(docker inspect --format {{.Id}} $CONTAINER_ID) sleep 90
                The bindfs /proc/$(docker inspect --format {{.State.Pid}} $CONTAINER_ID)/root /foo command resulting in /foo point to host file system (host root folder). Why? Thanks!
– Eduardo Lucio
                Mar 29, 2018 at 21:39
                I have the same problem on RHEL; note that Eduardo has already raised a (unanswered) question stackoverflow.com/questions/49565598/…
– Radim Vansa
                Jun 29, 2018 at 15:44
        
Thanks for contributing an answer to Stack Overflow!
Please be sure to answer the question. Provide details and share your research!
But avoid …
Asking for help, clarification, or responding to other answers.
Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.