Sysdig

Garbage In, Garbage Out
syscalls syscalls
*may contain traces of signals and shared memory

Syscalls
ssize_t read(int fd, void *buf, size_t count);
ssize_t write(int fd, const void *buf, size_t count);
int open(const char *pathname, int flags, mode_t mode);
int close(int fd);
int stat(const char *path, struct stat *buf);
...
!
$ grep -c __NR_ /usr/include/asm/unistd_64.h
313
!
$ man 2 read

strace
# strace cat /etc/hostname
execve("/bin/cat", ["cat", "/etc/hostname"], ...
brk(0) = 0x1675000
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT
mmap(NULL, 8192, PROT_READ|PROT_WRITE, ...
access("/etc/ld.so.preload", R_OK) = -1 ENOENT
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=48577, ...}) = 0
mmap(NULL, 48577, PROT_READ, MAP_PRIVATE, 3, 0) = ...
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT
(...)

that’s cool, but…
1 m illion sysca lls, as fast as possible
worst case for a ny tracer
# dd if=/dev/zero of=/dev/null bs=1k count=1M
1048576+0 records in
1048576+0 records out
1073741824 bytes (1.1 GB) copied, 0.332905 s, 3.2 GB/s
# strace -o /dev/null !!
1073741824 bytes (1.1 GB) copied, 18.2365 s, 58.9 MB/s
50x overhead

@brendangregg’s
http://www.slideshare.net/brendangregg/linux-performance-tools-2014

hello, sysdig
# sysdig | head -5
3 15:26:36.552482922 0 sysdig (7311) > switch
next=329(systemd-udevd) pgft_maj=6 pgft_min=1432
vm_size=26740 vm_rss=3052 vm_swap=0
4 15:26:36.552502349 0 systemd-udevd (329) < read
res=2352 data=# This file is part of systemd..#.#
systemd is free software; you can redistri
5 15:26:36.552590722 0 systemd-udevd (329) > read
fd=12(<f>/lib/udev/rules.d/42-usb-hid-pm.rules)
size=4096
res=0 data=
7 15:26:36.552596220 0 systemd-udevd (329) > close

hello, sysdig
# sysdig | head -5
3 15:26:36.552482922 0 sysdig (7311) > switch
size=4096
res=0 data=
event number, timestamp

hello, sysdig
# sysdig | head -5
3 15:26:36.552482922 0 sysdig (7311) > switch
size=4096
res=0 data=
CPU number, process name, pid

hello, sysdig
# sysdig | head -5
3 15:26:36.552482922 0 sysdig (7311) > switch
size=4096
res=0 data=
event direction, event type

hello, sysdig
enter event
# sysdig | head -5
3 15:26:36.552482922 0 sysdig (7311) > switch
size=4096
res=0 data=
exit event
syscall

hello, sysdig
# sysdig | head -5
3 15:26:36.552482922 0 sysdig (7311) > switch
size=4096
res=0 data=
arbitrary event attributes

events
# sysdig -L
(...)
> open()
< open(FD fd, FSPATH name, FLAGS32 flags, UINT32
mode)
> close(FD fd)
< close(ERRNO res)
> read(FD fd, UINT32 size)
< read(ERRNO res, BYTEBUF data)
> write(FD fd, UINT32 size)
< write(ERRNO res, BYTEBUF data)
(...)

dd
kernel

dd
kernel
strace
1073741824 bytes (1.1 GB) copied, 18.2365 s, 58.9 MB/s

dd
kernel
sysdig
ring buffer
1073741824 bytes (1.1 GB) copied, 1.30029 s, 826 MB/s

filters
fd.name FD full name. If the fd is a file, this
field contains the full path. If the FD
is a socket, this field contain the
connection tuple.
!
proc.apid the pid of one of the process
ancestors.
!
evt.latency delta between an exit event and the
correspondent enter event.
!
(...)
!
# sysdig -l | grep -Ec '^[a-z0-9_.]+'
88

filters
# sysdig fd.name contains shadow
2303 17:30:34.645573185 0 cat (24012) < open
fd=-13(EACCES) name=/etc/shadow flags=1(O_RDONLY)
mode=0
!
# sysdig evt.res = EACCES or evt.res = EPERM
617 17:32:16.197820784 0 cat (24027) < open
fd=-13(EACCES) name=/etc/shadow flags=1(O_RDONLY)
mode=0
4333 17:32:26.239052264 0 killall (24028) < kill
res=-1(EPERM)

back to that dd again…
# sysdig proc.name=not_dd > /dev/null & dd if=/dev/
zero of=/dev/null bs=1k count=1M ; killall sysdig
[1] 24070

output formatting
sa m e as filters (mostly)
# sysdig -p '%user.name %proc.name %fd.name: %evt.res'
evt.failed = true
ubuntu cat /etc/shadow: EACCES
ubuntu cat /usr/share/locale/en_US.UTF-8/LC_MESSAGES/
libc.mo: ENOENT
ubuntu cat /usr/share/locale/en_US.utf8/LC_MESSAGES/
libc.mo: ENOENT
ubuntu cat /usr/share/locale/en_US/LC_MESSAGES/
libc.mo: ENOENT

bottleneck in a haystack
# sysdig -p '%evt.latency.s.%evt.latency.ns %evt.dir
%evt.type %fd.name' fd.type contains ip and fd.sport != 22
(...)
0.000000000 >sendto 192.168.1.118:36220->46.28.247.84:80
0.000114365 <sendto 192.168.1.118:36220->46.28.247.84:80
0.000000000 >recvfrom 192.168.1.118:36220->46.28.247.84:80
0.000005090 <recvfrom 192.168.1.118:36220->46.28.247.84:80
0.000000000 >close 192.168.1.118:36220->46.28.247.84:80
0.000001587 <close 192.168.1.118:36220->46.28.247.84:80

sysdig -w
sysdig -r
sysdig -r
sysdig -r
.scap file
shit’s on fire, yo
capture trace file,
restore service analyze trace at your leisure

lies, damn lies and benchmarks
sysdig -w
sysdig -r
sysdig -r
sysdig -r
.scap file
do a single
benchmark run
analyze/postprocess
lots of ways

chisels: higher level of awesome
Lua
sysdig -cl
sysdig -i chisel_name
sysdig -c chisel_name [args…]

chisel all the things!
# sysdig -cl | grep -c ^[a-z]
37
# find /usr/share/sysdig/chisels/ -name '*.lua' | wc -l
42
the extra ones a re utilities to use in ch isels
(json, A NSI term ina l, etc.)

chisels: performance
bottlenecks Slowest system calls
fileslower Trace slow file I/O
netlower Trace slow network I/O
proc_exec_time Show process execution time
scallslower Trace slow syscalls
topscalls Top system calls by number of calls
topscalls_time Top system calls by time
yu p, a ty po ;)

chisels: security
list_login_shells List the login shell IDs
!
shellshock_detect print shellshock attacks
!
spy_users Display interactive user activity
power corru pts,
absolute power is even more fun

All right gentlemen,
we need some system info
lsof, ps, n etstat
lsof, ps, netstat
with time travel
http://draios.com/ps-lsof-netstat-time-travel/

version 0.1.91
do you feel lucky?
• some syscalls not yet implemented (no args)
• it did crash once (fixed immediately though)
• PID namespaces ignored
• root/privileged user only
• one sysdig process at a time
way better tha n strace though

Sysdig

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Sysdig

Similar to Sysdig (20)

More from gnosek

More from gnosek (8)

Recently uploaded

Recently uploaded (20)

Sysdig