devtools/taldump-analyze.py: script to analyze tal_dump() output.

Example output on my production machine, running v0.10.2rc1-9-g3d8293d:

```
Top sizes by name
backtrace: 1172021248 bytes, 4578208 items
wallet/txfilter.c:111:struct bitcoin_outpoint: 49584636 bytes, 1377351 items
lightningd/plugin.c:1651:char[]: 17261056 bytes, 16 items
lightningd/log.c:282:char[]: 16653908 bytes, 838086 items
wally_tal: 13390257 bytes, 122503 items
lightningd/log.c:250:struct log_entry[]: 7340032 bytes, 1 items
lightningd/jsonrpc.c:939:char[]: 6979648 bytes, 214 items
lightningd/log.c:440:u8[]: 3278359 bytes, 1338 items
common/memleak.c:51:struct memleak_notleak: 2216763 bytes, 2216763 items
Top sizes by path
['']: 2579337788 bytes, 1 items
['', 'lightningd/lightningd.c:103:struct lightningd']: 831485962 bytes, 1 items
['', 'lightningd/lightningd.c:103:struct lightningd', 'wallet/wallet.c:77:struct wallet']: 756169631 bytes, 1 items
['', 'lightningd/lightningd.c:103:struct lightningd', 'wallet/wallet.c:77:struct wallet', 'wallet/txfilter.c:133:struct outpointfilter']: 756166227 bytes, 2 items
['', 'lightningd/lightningd.c:103:struct lightningd', 'wallet/wallet.c:77:struct wallet', 'wallet/txfilter.c:133:struct outpointfilter', 'wallet/txfilter.c:134:struct outpointset']: 706581191 bytes, 2 items
['', 'ccan/ccan/tal/link/link.c:40:struct linkable']: 457580181 bytes, 1 items
['', 'ccan/ccan/tal/link/link.c:40:struct linkable', 'lightningd/log.c:236:struct log_book']: 430307262 bytes, 1 items
['', 'lightningd/lightningd.c:103:struct lightningd', 'lightningd/chaintopology.c:1061:struct chain_topology']: 48054597 bytes, 1 items
['', 'lightningd/lightningd.c:103:struct lightningd', 'lightningd/chaintopology.c:1061:struct chain_topology', 'lightningd/chaintopology.c:853:struct block']: 47076437 bytes, 1 items
['', 'lightningd/lightningd.c:103:struct lightningd', 'lightningd/chaintopology.c:1061:struct chain_topology', 'lightningd/chaintopology.c:853:struct block', 'bitcoin/block.c:210:struct bitcoin_tx *[]']: 47034285 bytes, 1 items
['', 'lightningd/lightningd.c:103:struct lightningd', 'lightningd/plugin.c:77:struct plugins']: 17569217 bytes, 1 items
['', 'lightningd/lightningd.c:103:struct lightningd', 'lightningd/plugin.c:77:struct plugins', 'lightningd/plugin.c:246:struct plugin', 'lightningd/plugin.c:1651:char[]']: 16784384 bytes, 5 items
['', 'lightningd/lightningd.c:103:struct lightningd', 'lightningd/chaintopology.c:1061:struct chain_topology', 'lightningd/chaintopology.c:853:struct block', 'bitcoin/block.c:210:struct bitcoin_tx *[]', 'bitcoin/tx.c:605:struct bitcoin_tx']: 13730127 bytes, 452 items
['', 'lightningd/lightningd.c:103:struct lightningd', 'lightningd/jsonrpc.c:1189:struct jsonrpc']: 7745890 bytes, 1 items
['', 'ccan/ccan/tal/link/link.c:40:struct linkable', 'lightningd/log.c:236:struct log_book', 'lightningd/log.c:250:struct log_entry[]']: 7340032 bytes, 1 items
['', 'lightningd/lightningd.c:103:struct lightningd', 'lightningd/chaintopology.c:1061:struct chain_topology', 'lightningd/chaintopology.c:853:struct block', 'bitcoin/block.c:210:struct bitcoin_tx *[]', 'bitcoin/tx.c:605:struct bitcoin_tx', 'wally_tal']: 4480328 bytes, 166 items
['', 'tmpctx']: 0 bytes, 1 items
```
This commit is contained in:
Rusty Russell 2021-11-24 14:36:03 +10:30
parent 8b348a5485
commit 1cbdb9cc12

73
devtools/taldump-analyze.py Executable file
View File

@ -0,0 +1,73 @@
#! /usr/bin/env python3
import sys
import re
# If you have names on their own lines, it's an old tal_dump which
# put a \n after children:
# awk '/CHILDREN/ { SAVED=$0 } !/CHILDREN/ { print SAVED$0; SAVED="" }' < lightningd-tal_dump > lightningd-tal_dump-children-on-same-line
# Dict of paths -> [total size,count]
by_path = {}
by_name = {}
SIZE_THRESHOLD = 1000000
def process(path, name, indent, bytelen):
# Finish any previous entries.
while indent < len(path):
# Don't track little ones
if by_path[','.join(path)][0] < SIZE_THRESHOLD:
del by_path[','.join(path)]
path = path[:-1]
# Add new child
assert indent == len(path)
# Parents must exist! Add bytes to all their tallies
for i in range(0, len(path)):
by_path[','.join(path[:i])][0] += bytelen
path += [name]
# Might already exist
prev = by_path.get(','.join(path), (0, 0))
by_path[','.join(path)] = [prev[0] + bytelen, prev[1] + 1]
prev = by_name.get(name, (0, 0))
by_name[name] = (prev[0] + bytelen, prev[1] + 1)
return path
cur_path = []
infile = open(sys.argv[1], 'rt')
while True:
line = infile.readline()
if not line:
break
stripped = line.lstrip(' ')
indent = (len(line) - len(stripped)) // 2
if not stripped.startswith('0x'):
print("Ignoring {}".format(line), end='')
continue
bytelen = int(re.search('len=([0-9]*)', line).group(1))
name = re.search('"(.*)"', line)
if not name:
# This can only happen at the root!
assert cur_path == []
name = ''
else:
name = name.group(1)
cur_path = process(cur_path, name, indent, bytelen)
print("Top sizes by name")
for k, v in sorted(by_name.items(), key=lambda x: -x[1][0]):
if v[0] < SIZE_THRESHOLD:
break
print("{}: {} bytes, {} items".format(k, v[0], v[1]))
print("Top sizes by path")
for k, v in sorted(by_path.items(), key=lambda x: -x[1][0]):
print("{}: {} bytes, {} items".format(k.split(','), v[0], v[1]))