Discussion about libperf

perf is the profiling and tracing tool available in the kernel source (tools/perf, tools/lib).

RAS stands for Reliability, Availability and Serviceability.

Detaild about the RAS daemon are at https://wiki.linaro.org/LEG/Engineering/Kernel/RAS/RAS_daemon.

This page describes the discussions around the changes needed to factor out the common code from perf and export is to other (new) tools. The first users would be perf and rasd.

Today it is needed to come to an agreement on how to move formward on the topic. The LKML discussion thread is progressing. Cf. https://lkml.org/lkml/2014/9/30/170.

Involved parties and contacts

IRC channel: #perf on irc.oftc.net

perf maintainers: Arnaldo, Borislav Petkov, Jiri Olsa, Ingo Molnar, Steven Rostedt etc.

Design Proposal: libperf

The work is scoped by LEG-386 as part of CARD-536.

The main design effort is to propose the API.

From an RFC/Proposal on LKML (https://lkml.org/lkml/2014/9/30/322), the common code between rasd and perf will moved, file by file, functionality by functionality.

A lot of replies later it seems that an agreement is found, cf. https://lkml.org/lkml/2014/10/7/338; and some patches are already flying, cf. https://lkml.org/lkml/2014/10/10/351.

From there the work consists of untagling the current implementation and move the minimum common API out of perf. Arnaldo (acme) has a branch with some clean-up as a preparation for the split up, cf. https://git.kernel.org/cgit/linux/kernel/git/acme/linux.git/log/?h=perf/hists.

Also a presentation 'Sharing kernel tools code' is planned at ELCE/LPC https://pdxplumbers.osuosl.org/2014/ocw/sessions/2511. Cf. the Etherpad notes at https://etherpad.fr/p/LPC2014_Tracing.

Date    Tue, 7 Oct 2014 10:40:21 -0300
From    Arnaldo Carvalho de Melo <>
Subject Re: perf & rasd integration plan
        

Em Tue, Oct 07, 2014 at 01:23:42PM +0200, Borislav Petkov escreveu:
> On Mon, Oct 06, 2014 at 06:22:43PM -0300, Arnaldo Carvalho de Melo wrote:
> > My preference would be for single object files, but the pressure to have
> > a written in stone library seems to just build up...

> Yeah, I think if we have an alternative way to easily integrate that
> functionality into external projects, we can avoid the need for a lib.

> ...

> > After doing a 'make perf-targz-src-pkg'

> > I.e. no kernel sources involved on the machines where I build test.

> > IOW, it is untangled from the kernel sources. As tools/lib/api/ should
> > as well.

> Ok, that actually sounds like something we should try:

> $ make perf-api-src-pkg

> which would spit a tgz with all the generic stuff perf carries. People
> would simply unpack it in their projects and build it.

> That would require somewhat stable function names so that updating is
> relatively painless and if visible functions change, some sort of a

Right, we now should always think that the great advantage of having
this code and its users in the same repo tree is going away, i.e. we
will not be able to improve the code by fixing mistakes that require
changing its users.

So we should export the bare minimum that tools advocate exporting, like
you guys are doing now by providing the code that shows how you use it.

> README file should say what and why.

Well, at that point we can't do that anymore, we will need instead to
add new methods while keeping the existing ones in place and with the
same semantics, at most for something really annoying to maintain we
will have to add annoying warnings telling that in a year or two
something will go away, it was deprecated, annoy the maintainer of your
tool to update their project.

> Sounds like a good alternative to a lib to me...

Yes, should be interesting to experiment with this, at the very least it
can be considered a step into the direction of a full blown library, if
we ever feel that it is strictly needed.

> > Well, the rasd experience is serving to show areas where there is
> > unnecessary entanglement (hists inside perf_event, etc, the ifdefs you
> > put in place).

> Yep.

> > I'm working to remove the ones that are in rasd.c, aiming to have a
> > tools/lib/api/ tree that can be used to build rasd and tools/perf/.

> > What I don't want to do is to simply straight more

       s/more/move/g

> > tools/perf/util/evlist.c to tools/lib/api/perf/, some untanglement work
> > is needed.

> Yep.

- Arnaldo

Date    Tue, 30 Sep 2014 11:06:21 +0200
Subject perf & rasd integration plan
From    Jean Pihet <>
        

The RAS Deamon (rasd) as available at [1] and the development version
at [2], allows to enable kernel tracepoints and outputs the
tracepoints fields according to the kernel format.

[1] https://git.kernel.org/cgit/utils/ras/rasd.git/
[2] https://git.linaro.org/people/jean.pihet/rasd.git

rasd currently is a duplicate of the perf tool code, ultimately perf
and rasd will use the same common code. The goal is to factor out the
common code from perf and place it in small librairies in tools/lib.

Here is the code that rasd currently uses and that should be moved to tools/lib:

- debugfs: already in tools/lib/api/fs
  . mount and retrieve path

- evlist: tools/perf/util/evlist.[ch]
  . create and init new evlist,
  . set cpu and thread maps,
  . add events to evlist,
  . init and use internal event id,
  . alloc and mmap events buffers, manage file descriptors,
  . enable events,
  . read events buffers, parse data,
  . unmap and free buffers

- evsel: tools/perf/util/evsel.[ch]
  . create and init new tracepoints events,
  . init and use internal event id,
  . open events, manage fds,
  . close and free events

- trace-event: tools/perf/util/trace-event.[ch] and
  tools/perf/util/trace-event-parse.c
  . retrieve and parse events format (using event-parse),
  . print out events fields

- event-parse: already in tools/lib/traceevent/event-parse.[ch]
  . retrieve and parse events format,
  . parse events format and print out events fields

- trace-seq: already in tools/lib/traceevent/trace-seq.c
  . format output string for event fields

- events plugins: already in tools/lib/traceevent/event-plugin.c

- util: tools/perf/util/util.[ch]
  . files open/read,
  . manage events attributes,
  . various macros

- test events attributes: tools/perf/tests/attr.c
  . test_attr__open()

- thread: tools/perf/util/thread_map.[ch] and
- cpu: tools/perf/util/cpumap.[ch]
  . init and manage process maps

- xyarray: tools/perf/util/xyarray.[ch]

- syscall: tools/perf/perf-sys.h

- cgroup: tools/perf/util/cgroup.[ch]

The plan is to move the small and generic functions first: util,
xyarray, cpumap, thread_map etc; then evlist, evsel, trace-event,
trace-event-parse; and finally integrate rasd into the tools/ dir.

Here is a task breakdown and estimates. Warning: the list is now (Oct 6 2014) obsolete after the discussions on LKML since a clean-up is first needed.

The problem with the proposed approach is that it is parctically impossible to give the dates for completion.

Tasks breakdown

Name

Man.Days

cpumap, thread_map

1

xyarray, cgroup

1

evlist

2

evsel

2

trace-event, trace-event-parse

2

util/util, attr

1

perf-sys

1

Total

10

Initial proposal, for reference

From Borislav Petkov's proposal on Fri, 15 Nov 2013:

From: Borislav Petkov <bp@alien8.de>
To: Ingo Molnar <mingo@kernel.org>, Arnaldo Carvalho de Melo
 <acme@infradead.org>
Cc: Robert Richter <rric@kernel.org>, Peter Zijlstra <peterz@infradead.org>,
 Jiri Olsa <jolsa@redhat.com>
Subject: Re: persistent events: Tooling support; perf tool splitup
Message-ID: <20131115202848.GK29277@pd.tnic>
User-Agent: Mutt/1.5.21 (2010-09-15)
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Content-Type: text/plain; charset=utf-8
Status: RO

Hi guys,

I'm finally ready with the next attempt at a carve out of the facilities
in tools/perf/ into libraries for external use. I'm adding the diffstat
of the patches below with comments so that you can see how it would look
like. Once we all agree on the structure, I'll start feeding acme 1-2
patches at a time so that there are no big merge issues.

Comments and suggestions are, as always, very welcome and needed. :-)

* The first group are the kernel headers any tools in tools/
could/are/would be using. I've put them into tools/include/ because it
leaves the #include <linux...> directives unchanged.

But since we're making a liblk kernel library, I can imagine putting
them into tools/lib/lk/ and making all part of that generic library.

 tools/{perf/util => }/include/asm/bug.h            |    6 +-
 tools/{perf/util => }/include/linux/bitmap.h       |    7 +-
 tools/{perf/util => }/include/linux/bitops.h       |   12 +-
 tools/{perf/util => }/include/linux/compiler.h     |    8 +-
 tools/{perf/util => }/include/linux/kernel.h       |    6 +-
 tools/{perf/util => }/include/linux/list.h         |    9 +-
 tools/{perf/util => }/include/linux/types.h        |    6 +-
 tools/include/asm/byteorder.h                      |    2 +
 tools/include/linux/export.h                       |    6 +
 tools/include/linux/hash.h                         |    8 +
 tools/include/linux/poison.h                       |    1 +
 tools/include/linux/prefetch.h                     |    6 +
 tools/include/linux/rbtree.h                       |    3 +
 tools/include/linux/rbtree_augmented.h             |    2 +

This is tools/lib/lk/liblk.a which contains only compilation units which
contain generic code. The idea behind this library is to be used by all
tools in tools/.

 tools/{perf/util => lib/lk}/color.c                |   69 +-
 tools/{perf/util => lib/lk}/color.h                |   34 +-
 tools/{perf/util => lib/lk}/cpumap.c               |    9 +-
 tools/{perf/util => lib/lk}/cpumap.h               |    4 +
 tools/{perf/util => lib/lk}/ctype.c                |    2 +-
 tools/{perf/util => lib/lk}/hweight.c              |    1 +
 tools/{perf/util => lib/lk}/include/linux/magic.h  |    0
 tools/{perf/util => lib/lk}/parse-options.c        |    6 +-
 tools/{perf/util => lib/lk}/parse-options.h        |   13 +-
 tools/{perf/util => lib/lk}/rblist.c               |    0
 tools/{perf/util => lib/lk}/rblist.h               |    6 +-
 tools/{perf/util => lib/lk}/strbuf.c               |    3 +-
 tools/{perf/util => lib/lk}/strbuf.h               |   15 +-
 tools/{perf/util => lib/lk}/string.c               |   42 +-
 tools/{perf/util => lib/lk}/strlist.c              |    0
 tools/{perf/util => lib/lk}/strlist.h              |    6 +-
 tools/{perf/util => lib/lk}/sysfs.c                |    4 +
 tools/{perf/util => lib/lk}/thread_map.c           |    0
 tools/{perf/util => lib/lk}/thread_map.h           |    6 +-
 tools/{perf/util => lib/lk}/types.h                |    6 +-
 tools/{perf/util => lib/lk}/usage.c                |    2 +-
 tools/{perf/util => lib/lk}/wrapper.c              |    3 +-
 tools/{perf/util => lib/lk}/xyarray.c              |    0
 tools/{perf/util => lib/lk}/xyarray.h              |    6 +-
 tools/lib/lk/Makefile                              |   62 +-
 tools/lib/lk/config.c                              |  370 ++++++
 tools/lib/lk/config.h                              |   13 +
 tools/lib/lk/ctype.h                               |   37 +
 tools/lib/lk/hweight.h                             |   13 +
 tools/lib/lk/string.h                              |   21 +
 tools/lib/lk/sysfs.h                               |   12 +
 tools/lib/lk/usage.h                               |   34 +
 tools/lib/lk/util.c                                |   18 +
 tools/lib/lk/util.h                                |   38 +
 tools/lib/lk/wrapper.h                             |   27 +

This is lib/perf/libperfevent.a which got initiated from exporting
evsel.* and evlist.* which I'm going to need for other tools to use.
Diffstat shows what moves in there, I've tried to keep it as small as
possible.

 tools/{perf/util => lib/perf}/callchain.c          |    4 +-
 tools/{perf/util => lib/perf}/callchain.h          |   12 +-
 tools/{perf/util => lib/perf}/cgroup.c             |    8 +-
 tools/{perf/util => lib/perf}/cgroup.h             |    7 +-
 tools/{perf/util => lib/perf}/debug.c              |   29 +-
 tools/{perf/util => lib/perf}/debug.h              |   19 +-
 tools/{perf/util => lib/perf}/dso.c                |    8 +-
 tools/{perf/util => lib/perf}/dso.h                |   11 +-
 tools/{perf/util => lib/perf}/event.c              |  109 +-
 tools/{perf/util => lib/perf}/event.h              |   12 +-
 tools/{perf/util => lib/perf}/evlist.c             |   37 +-
 tools/{perf/util => lib/perf}/evlist.h             |   15 +-
 tools/{perf/util => lib/perf}/evsel.c              |   38 +-
 tools/{perf/util => lib/perf}/evsel.h              |   33 +-
 tools/{perf/util => lib/perf}/map.c                |   10 +-
 tools/{perf/util => lib/perf}/map.h                |    9 +-
 tools/{perf/util => lib/perf}/path.c               |   28 +-
 tools/{perf/util => lib/perf}/perf_regs.h          |    6 +-
 tools/{perf/util => lib/perf}/symbol.c             |    6 +-
 tools/{perf/util => lib/perf}/symbol.h             |   10 +-
 tools/{perf/util => lib/perf}/target.c             |    3 -
 tools/{perf/util => lib/perf}/target.h             |   12 +-
 tools/{perf/util => lib/perf}/thread.h             |    7 +-
 tools/{perf/util => lib/perf}/tool.h               |    6 +-
 tools/{perf/util => lib/perf}/vdso.c               |    4 +-
 tools/{perf/util => lib/perf}/vdso.h               |    6 +-
 tools/lib/perf/Makefile                            |   83 ++
 tools/lib/perf/arch.h                              |  112 ++
 tools/lib/perf/build-id.c                          |  102 ++
 tools/lib/perf/build-id.h                          |   32 +
 tools/lib/perf/hist.h                              |   69 ++
 tools/lib/perf/machine.c                           | 1268 ++++++++++++++++++++
 tools/lib/perf/machine.h                           |  168 +++
 tools/lib/perf/path.h                              |   29 +
 tools/lib/perf/perf.c                              |   29 +
 tools/lib/perf/perf.h                              |  105 ++
 tools/lib/perf/thread.c                            |  187 +++

Rest is changes to tools/perf/

 tools/perf/MANIFEST                                |    3 -
 tools/perf/Makefile                                |  136 +--
 tools/perf/arch/common.c                           |    4 +-
 tools/perf/arch/x86/include/perf_regs.h            |    2 +-
 tools/perf/arch/x86/util/header.c                  |    2 +-
 tools/perf/arch/x86/util/tsc.c                     |    4 +-
 tools/perf/arch/x86/util/tsc.h                     |    2 +-
 tools/perf/bench/mem-memcpy.c                      |    8 +-
 tools/perf/bench/mem-memset.c                      |    8 +-
 tools/perf/bench/numa.c                            |    6 +-
 tools/perf/bench/sched-messaging.c                 |    4 +-
 tools/perf/bench/sched-pipe.c                      |    4 +-
 tools/perf/builtin-annotate.c                      |   21 +-
 tools/perf/builtin-bench.c                         |    4 +-
 tools/perf/builtin-buildid-cache.c                 |   12 +-
 tools/perf/builtin-buildid-list.c                  |   19 +-
 tools/perf/builtin-diff.c                          |   14 +-
 tools/perf/builtin-evlist.c                        |    8 +-
 tools/perf/builtin-help.c                          |   15 +-
 tools/perf/builtin-inject.c                        |   18 +-
 tools/perf/builtin-kmem.c                          |   16 +-
 tools/perf/builtin-kvm.c                           |   20 +-
 tools/perf/builtin-lock.c                          |   20 +-
 tools/perf/builtin-mem.c                           |    6 +-
 tools/perf/builtin-probe.c                         |   10 +-
 tools/perf/builtin-record.c                        |  162 ++-
 tools/perf/builtin-report.c                        |   26 +-
 tools/perf/builtin-sched.c                         |   21 +-
 tools/perf/builtin-script.c                        |   18 +-
 tools/perf/builtin-stat.c                          |   29 +-
 tools/perf/builtin-timechart.c                     |   21 +-
 tools/perf/builtin-top.c                           |   27 +-
 tools/perf/builtin-trace.c                         |   41 +-
 tools/perf/builtin.h                               |    2 +-
 tools/perf/config/Makefile                         |    4 +-
 tools/perf/perf.c                                  |   18 +-
 tools/perf/perf.h                                  |  187 +--
 tools/perf/tests/attr.c                            |    5 +-
 tools/perf/tests/bp_signal.c                       |    2 +-
 tools/perf/tests/bp_signal_overflow.c              |    2 +-
 tools/perf/tests/builtin-test.c                    |   15 +-
 tools/perf/tests/code-reading.c                    |   14 +-
 tools/perf/tests/dso-data.c                        |    6 +-
 tools/perf/tests/evsel-roundtrip-name.c            |    6 +-
 tools/perf/tests/evsel-tp-sched.c                  |    2 +-
 tools/perf/tests/hists_link.c                      |   12 +-
 tools/perf/tests/keep-tracking.c                   |    8 +-
 tools/perf/tests/mmap-basic.c                      |    9 +-
 tools/perf/tests/open-syscall-all-cpus.c           |    8 +-
 tools/perf/tests/open-syscall-tp-fields.c          |    6 +-
 tools/perf/tests/open-syscall.c                    |    6 +-
 tools/perf/tests/parse-events.c                    |    9 +-
 tools/perf/tests/parse-no-sample-id-all.c          |    4 +-
 tools/perf/tests/perf-record.c                     |    6 +-
 tools/perf/tests/perf-time-to-tsc.c                |    8 +-
 tools/perf/tests/python-use.c                      |    2 -
 tools/perf/tests/rdpmc.c                           |    4 +-
 tools/perf/tests/sample-parsing.c                  |    5 +-
 tools/perf/tests/sw-clock.c                        |    8 +-
 tools/perf/tests/task-exit.c                       |    8 +-
 tools/perf/tests/tests.h                           |    4 +
 tools/perf/tests/vmlinux-kallsyms.c                |    7 +-
 tools/perf/ui/browser.c                            |    2 +-
 tools/perf/ui/browser.h                            |    2 +-
 tools/perf/ui/browsers/annotate.c                  |    8 +-
 tools/perf/ui/browsers/hists.c                     |    6 +-
 tools/perf/ui/browsers/map.c                       |    4 +-
 tools/perf/ui/browsers/scripts.c                   |    7 +-
 tools/perf/ui/gtk/annotate.c                       |    4 +-
 tools/perf/ui/gtk/browser.c                        |    6 +-
 tools/perf/ui/gtk/helpline.c                       |    2 +-
 tools/perf/ui/gtk/hists.c                          |    6 +-
 tools/perf/ui/gtk/setup.c                          |    2 +-
 tools/perf/ui/gtk/util.c                           |    2 +-
 tools/perf/ui/helpline.c                           |    2 +-
 tools/perf/ui/helpline.h                           |    1 -
 tools/perf/ui/hist.c                               |    4 +-
 tools/perf/ui/progress.h                           |    2 +-
 tools/perf/ui/setup.c                              |    4 +-
 tools/perf/ui/stdio/hist.c                         |    4 +-
 tools/perf/ui/tui/helpline.c                       |    2 +-
 tools/perf/ui/tui/setup.c                          |    3 +-
 tools/perf/ui/tui/util.c                           |    2 +-
 tools/perf/ui/ui.h                                 |    4 +-
 tools/perf/ui/util.c                               |    2 +-
 tools/perf/util/abspath.c                          |    2 +
 tools/perf/util/alias.c                            |    2 +-
 tools/perf/util/annotate.c                         |   14 +-
 tools/perf/util/annotate.h                         |    4 +-
 tools/perf/util/build-id.c                         |  129 +-
 tools/perf/util/build-id.h                         |   24 +-
 tools/perf/util/cache.h                            |   45 +-
 tools/perf/util/config.c                           |  463 +------
 tools/perf/util/dwarf-aux.c                        |    2 +-
 tools/perf/util/exec_cmd.c                         |   10 +-
 tools/perf/util/exec_cmd.h                         |    6 +-
 tools/perf/util/header.c                           |   30 +-
 tools/perf/util/header.h                           |   16 +-
 tools/perf/util/help.c                             |    2 +-
 tools/perf/util/hist.c                             |   70 +-
 tools/perf/util/hist.h                             |  195 ++-
 tools/perf/util/include/asm/byteorder.h            |    2 -
 tools/perf/util/include/asm/hweight.h              |    8 -
 tools/perf/util/include/linux/export.h             |    6 -
 tools/perf/util/include/linux/hash.h               |    5 -
 tools/perf/util/include/linux/poison.h             |    1 -
 tools/perf/util/include/linux/prefetch.h           |    6 -
 tools/perf/util/include/linux/rbtree.h             |    2 -
 tools/perf/util/include/linux/rbtree_augmented.h   |    2 -
 tools/perf/util/include/linux/string.h             |    4 -
 tools/perf/util/intlist.h                          |    2 +-
 tools/perf/util/machine.c                          | 1258 +------------------
 tools/perf/util/machine.h                          |  163 +--
 tools/perf/util/pager.c                            |   15 +-
 tools/perf/util/parse-events.c                     |   12 +-
 tools/perf/util/parse-events.h                     |    3 +-
 tools/perf/util/parse-events.y                     |    2 +-
 tools/perf/util/pmu.c                              |    4 +-
 tools/perf/util/probe-event.c                      |   14 +-
 tools/perf/util/probe-event.h                      |    2 +-
 tools/perf/util/probe-finder.c                     |    7 +-
 tools/perf/util/probe-finder.h                     |    2 +
 tools/perf/util/python-ext-sources                 |   22 +-
 tools/perf/util/python.c                           |   12 +-
 tools/perf/util/record.c                           |    8 +-
 .../perf/util/scripting-engines/trace-event-perl.c |    6 +-
 .../util/scripting-engines/trace-event-python.c    |    6 +-
 tools/perf/util/session.c                          |   17 +-
 tools/perf/util/session.h                          |    8 +-
 tools/perf/util/setup.py                           |    3 +-
 tools/perf/util/sort.c                             |    8 +-
 tools/perf/util/sort.h                             |   83 +-
 tools/perf/util/stat.h                             |    2 +-
 tools/perf/util/strfilter.c                        |    2 +
 tools/perf/util/svghelper.h                        |    2 +-
 tools/perf/util/symbol-elf.c                       |    4 +-
 tools/perf/util/symbol-minimal.c                   |    2 +-
 tools/perf/util/sysfs.h                            |    6 -
 tools/perf/util/thread.c                           |   91 --
 tools/perf/util/top.c                              |   10 +-
 tools/perf/util/top.h                              |    6 +-
 tools/perf/util/trace-event-info.c                 |    2 +-
 tools/perf/util/trace-event-read.c                 |    2 -
 tools/perf/util/unwind.c                           |    4 +-
 tools/perf/util/unwind.h                           |    6 +-
 tools/perf/util/util.c                             |   98 +-
 tools/perf/util/util.h                             |  193 ++-
 tools/perf/util/values.h                           |    2 +-
 238 files changed, 4299 insertions(+), 3722 deletions(-)

LEG/ServerArchitecture/libperfForPerfAndRAS (last modified 2017-08-17 12:12:45)