Merge branch 'Gallopsled:dev' into dev

CHANGELOG.md

@@ -11,6 +11,7 @@ The table below shows which release corresponds to each branch, and what date th
 | ---------------- | -------- | ---------------------- |
 | [4.14.0](#4140-dev)  | `dev`   |
 | [4.13.0](#4130-beta)  | `beta`    |
+| [4.12.1](#4121)  |          |
 | [4.12.0](#4120-stable)  | `stable`   | Feb 22, 2024
 | [4.11.1](#4111)  |          | Nov 14, 2023
 | [4.11.0](#4110)  |          | Sep 15, 2023
@@ -78,13 +79,21 @@ The table below shows which release corresponds to each branch, and what date th
 - [#2327][2327] Add basic support to debug processes on Windows
 - [#2322][2322] Add basic RISCV64 shellcraft support
 - [#2330][2330] Change `context.newline` when setting `context.os` to `"windows"`
+- [#2389][2389] Fix passing bytes to `context.log_file` and `crc.BitPolynom`
+- [#2391][2391] Fix error message when passing invalid kwargs to `xor`
+- [#2376][2376] Return buffered data on first EOF in tube.readline()
+- [#2387][2387] Convert apport_corefile() output from bytes-like object to string
 
 [2360]: https://github.com/Gallopsled/pwntools/pull/2360
 [2356]: https://github.com/Gallopsled/pwntools/pull/2356
 [2374]: https://github.com/Gallopsled/pwntools/pull/2374
 [2327]: https://github.com/Gallopsled/pwntools/pull/2327
 [2322]: https://github.com/Gallopsled/pwntools/pull/2322
 [2330]: https://github.com/Gallopsled/pwntools/pull/2330
+[2389]: https://github.com/Gallopsled/pwntools/pull/2389
+[2391]: https://github.com/Gallopsled/pwntools/pull/2391
+[2376]: https://github.com/Gallopsled/pwntools/pull/2376
+[2387]: https://github.com/Gallopsled/pwntools/pull/2387
 
 ## 4.13.0 (`beta`)
 
@@ -143,6 +152,14 @@ The table below shows which release corresponds to each branch, and what date th
 [2347]: https://github.com/Gallopsled/pwntools/pull/2347
 [2233]: https://github.com/Gallopsled/pwntools/pull/2233
 
+## 4.12.1
+
+- [#2373][2373] Fix displaying bright color variation in terminal output
+- [#2378][2378] Don't go though a shell in `gdb.debug`
+
+[2373]: https://github.com/Gallopsled/pwntools/pull/2373
+[2378]: https://github.com/Gallopsled/pwntools/pull/2378
+
 ## 4.12.0 (`stable`)
 
 - [#2202][2202] Fix `remote` and `listen` in sagemath

README.md

@@ -37,8 +37,8 @@ Pwntools is best supported on 64-bit Ubuntu LTS releases (14.04, 16.04, 18.04, a
 Python3 is suggested, but Pwntools still works with Python 2.7.  Most of the functionality of pwntools is self-contained and Python-only.  You should be able to get running quickly with
 
 ```sh
-apt-get update
-apt-get install python3 python3-pip python3-dev git libssl-dev libffi-dev build-essential
+sudo apt-get update
+sudo apt-get install python3 python3-pip python3-dev git libssl-dev libffi-dev build-essential
 python3 -m pip install --upgrade pip
 python3 -m pip install --upgrade pwntools
 ```

docs/source/install.rst

@@ -15,6 +15,9 @@ following system libraries installed.
 
    install/*
 
+
+Note: For Mac OS X you will need to have cmake ``brew install cmake`` and pkg-config ``brew install pkg-config`` installed.
+
 Released Version
 -----------------
 
@@ -25,8 +28,8 @@ Python3
 
 .. code-block:: bash
 
-    $ apt-get update
-    $ apt-get install python3 python3-pip python3-dev git libssl-dev libffi-dev build-essential
+    $ sudo apt-get update
+    $ sudo apt-get install python3 python3-pip python3-dev git libssl-dev libffi-dev build-essential
     $ python3 -m pip install --upgrade pip
     $ python3 -m pip install --upgrade pwntools
 
@@ -40,8 +43,8 @@ Additionally, due to `pip` dropping support for Python2, a specfic version of `p
 
 .. code-block:: bash
 
-    $ apt-get update
-    $ apt-get install python python-pip python-dev git libssl-dev libffi-dev build-essential
+    $ sudo apt-get update
+    $ sudo apt-get install python python-pip python-dev git libssl-dev libffi-dev build-essential
     $ python2 -m pip install --upgrade pip==20.3.4
     $ python2 -m pip install --upgrade pwntools
 

docs/source/install/binutils.rst

@@ -18,28 +18,29 @@ Ubuntu Xenial (16.04) has official packages for most architectures, and does not
 
 .. code-block:: bash
 
-    $ apt-get install software-properties-common
-    $ apt-add-repository ppa:pwntools/binutils
-    $ apt-get update
+    $ sudo apt-get install software-properties-common
+    $ sudo apt-add-repository ppa:pwntools/binutils
+    $ sudo apt-get update
 
 Then, install the binutils for your architecture.
 
 .. code-block:: bash
 
-    $ apt-get install binutils-$ARCH-linux-gnu
+    $ sudo apt-get install binutils-$ARCH-linux-gnu
 
 Mac OS X
 ^^^^^^^^^^^^^^^^
 
 Mac OS X is just as easy, but requires building binutils from source.
-However, we've made ``homebrew`` recipes to make this a single command.
+However, we've made ``homebrew`` recipes to make this just two commands.
 After installing `brew <https://brew.sh>`__, grab the appropriate
 recipe from our `binutils
 repo <https://github.com/Gallopsled/pwntools-binutils/>`__.
 
 .. code-block:: bash
 
-    $ brew install https://raw.githubusercontent.com/Gallopsled/pwntools-binutils/master/macos/binutils-$ARCH.rb
+    $ wget https://raw.githubusercontent.com/Gallopsled/pwntools-binutils/master/macos/binutils-$ARCH.rb
+    $ brew install ./binutils-$ARCH.rb
 
 Alternate OSes
 ^^^^^^^^^^^^^^^^

docs/source/install/headers.rst

@@ -10,7 +10,7 @@ Ubuntu
 
 .. code-block:: bash
 
-    $ apt-get install python-dev
+    $ sudo apt-get install python-dev
 
 Mac OS X
 ^^^^^^^^^^^^^^^^

pwnlib/context/__init__.py

@@ -367,6 +367,7 @@ class ContextType(object):
         'randomize': False,
         'rename_corefiles': True,
         'newline': b'\n',
+        'throw_eof_on_incomplete_line': None,
         'noptrace': False,
         'os': 'linux',
         'proxy': None,
@@ -1033,6 +1034,8 @@ def log_file(self, value):
         """
         if isinstance(value, (bytes, six.text_type)):
             # check if mode was specified as "[value],[mode]"
+            from pwnlib.util.packing import _need_text
+            value = _need_text(value)
             if ',' not in value:
                 value += ',a'
             filename, mode = value.rsplit(',', 1)
@@ -1488,6 +1491,25 @@ def newline(self, v):
         # circular imports
         from pwnlib.util.packing import _need_bytes
         return _need_bytes(v)
+
+    @_validator
+    def throw_eof_on_incomplete_line(self, v):
+        """Whether to raise an :class:`EOFError` if an EOF is received before a newline in ``tube.recvline``.
+
+        Controls if an :class:`EOFError` is treated as newline in ``tube.recvline`` and similar functions
+        and whether a warning should be logged about it.
+
+        Possible values are:
+
+        - ``True``: Raise an :class:`EOFError` if an EOF is received before a newline.
+        - ``False``: Return the data received so far if an EOF is received
+          before a newline without logging a warning.
+        - ``None``: Return the data received so far if an EOF is received
+          before a newline and log a warning.
+
+        Default value is ``None``.
+        """
+        return v if v is None else bool(v)
 
 
     @_validator

pwnlib/elf/corefile.py

@@ -1329,6 +1329,10 @@ def apport_read_crash_data(self):
         except Exception:
             pass
 
+        # Convert bytes-like object to string
+        if isinstance(data, bytes):
+            data = data.decode('utf-8')
+
         return data
 
     def systemd_coredump_corefile(self):

pwnlib/gdb.py

@@ -353,6 +353,10 @@ def _gdbserver_args(pid=None, path=None, port=0, gdbserver_args=None, args=None,
         gdbserver_args += ['--wrapper', python_wrapper_script, '--']
     elif env is not None:
         gdbserver_args += ['--wrapper', which('env'), '-i'] + env_args + ['--']
+    # --no-startup-with-shell is required for forking shells like SHELL=/bin/fish
+    # https://github.com/Gallopsled/pwntools/issues/2377
+    else:
+        gdbserver_args += ['--no-startup-with-shell']
 
     gdbserver_args += ['localhost:%d' % port]
     gdbserver_args += args

pwnlib/term/text.py

@@ -27,9 +27,7 @@ class Module(types.ModuleType):
     def __init__(self):
         self.__file__ = __file__
         self.__name__ = __name__
-        self.num_colors = 8
-        self.has_bright = self.num_colors >= 16
-        self.has_gray = self.has_bright
+        self.num_colors = termcap.get('colors', 8) if sys.platform == 'win32' else 8
         self.when = 'auto'
         self._colors = {
             'black': 0,
@@ -61,6 +59,14 @@ def when(self):
     def when(self, val):
         self._when = eval_when(val)
 
+    @property
+    def has_bright(self):
+        return self.num_colors >= 16
+
+    @property
+    def has_gray(self):
+        return self.has_bright
+
     def _fg_color(self, c):
         c = termcap.get('setaf', c) or termcap.get('setf', c)
         if not hasattr(c, 'encode'):

pwnlib/tubes/ssh.py

@@ -544,20 +544,32 @@ class ssh(Timeout, Logger):
     #: Remote port (``int``)
     port = None
 
+    #: Remote username (``str``)
+    user = None
+
+    #: Remote password (``str``)
+    password = None
+
+    #: Remote private key (``str``)
+    key = None
+
+    #: Remote private key file (``str``)
+    keyfile = None
+
     #: Enable caching of SSH downloads (``bool``)
     cache = True
 
+    #: Enable raw mode and don't probe the environment (``bool``)
+    raw = False
+
     #: Paramiko SSHClient which backs this object
     client = None
 
-    #: Paramiko SFTPClient object which is used for file transfers.
-    #: Set to :const:`None` to disable ``sftp``.
-    sftp = None
-
     #: PID of the remote ``sshd`` process servicing this connection.
     pid = None
 
     _cwd = '.'
+    _tried_sftp = False
 
     def __init__(self, user=None, host=None, port=22, password=None, key=None,
                  keyfile=None, proxy_command=None, proxy_sock=None, level=None,
@@ -719,6 +731,9 @@ def cwd(self, cwd):
 
     @property
     def sftp(self):
+        """Paramiko SFTPClient object which is used for file transfers.
+        Set to :const:`None` to disable ``sftp``.
+        """
         if not self._tried_sftp:
             try:
                 self._sftp = self.transport.open_sftp_client()

pwnlib/tubes/tube.py

@@ -467,19 +467,31 @@ def recvline(self, keepends=True, timeout=default):
         Receive a single line from the tube.
 
         A "line" is any sequence of bytes terminated by the byte sequence
-        set in :attr:`newline`, which defaults to ``'\n'``.
+        set in :attr:`newline`, which defaults to ``b'\n'``.
+
+        If the connection is closed (:class:`EOFError`) before a newline
+        is received, the buffered data is returned by default and a warning
+        is logged. If the buffer is empty, an :class:`EOFError` is raised.
+        This behavior can be changed by setting :meth:`pwnlib.context.ContextType.throw_eof_on_incomplete_line`.
 
         If the request is not satisfied before ``timeout`` seconds pass,
-        all data is buffered and an empty string (``''``) is returned.
+        all data is buffered and an empty byte string (``b''``) is returned.
 
         Arguments:
             keepends(bool): Keep the line ending (:const:`True`).
             timeout(int): Timeout
 
+        Raises:
+            :class:`EOFError`: The connection closed before the request
+                                 could be satisfied and the buffer is empty
+
         Return:
             All bytes received over the tube until the first
             newline ``'\n'`` is received.  Optionally retains
-            the ending.
+            the ending. If the connection is closed before a newline
+            is received, the remaining data received up to this point
+            is returned.
+
 
         Examples:
 
@@ -494,8 +506,31 @@ def recvline(self, keepends=True, timeout=default):
             >>> t.newline = b'\r\n'
             >>> t.recvline(keepends = False)
             b'Foo\nBar'
+            >>> t = tube()
+            >>> def _recv_eof(n):
+            ...     if not _recv_eof.throw:
+            ...         _recv_eof.throw = True
+            ...         return b'real line\ntrailing data'
+            ...     raise EOFError
+            >>> _recv_eof.throw = False
+            >>> t.recv_raw = _recv_eof
+            >>> t.recvline()
+            b'real line\n'
+            >>> t.recvline()
+            b'trailing data'
+            >>> t.recvline() # doctest: +ELLIPSIS
+            Traceback (most recent call last):
+            ...
+            EOFError
         """
-        return self.recvuntil(self.newline, drop = not keepends, timeout = timeout)
+        try:
+            return self.recvuntil(self.newline, drop = not keepends, timeout = timeout)
+        except EOFError:
+            if not context.throw_eof_on_incomplete_line and self.buffer.size > 0:
+                if context.throw_eof_on_incomplete_line is None:
+                    self.warn_once('EOFError during recvline. Returning buffered data without trailing newline.')
+                return self.buffer.get()
+            raise
 
     def recvline_pred(self, pred, keepends=False, timeout=default):
         r"""recvline_pred(pred, keepends=False) -> bytes

pwnlib/util/crc/__init__.py

@@ -74,6 +74,8 @@ class BitPolynom(object):
 
     def __init__(self, n):
         if isinstance(n, (bytes, six.text_type)):
+            from pwnlib.util.packing import _need_text
+            n = _need_text(n)
             self.n = 0
             x = BitPolynom(2)
             try:

pwnlib/util/fiddling.py

@@ -320,14 +320,19 @@ def xor(*args, **kwargs):
        The string of the arguments xor'ed together.
 
     Example:
+
        >>> xor(b'lol', b'hello', 42)
        b'. ***'
+       >>> xor(cut = 'min', other = '')
+       Traceback (most recent call last):
+         ...
+       TypeError: xor() got an unexpected keyword argument 'other'
     """
 
     cut = kwargs.pop('cut', 'max')
 
     if kwargs != {}:
-        raise TypeError("xor() got an unexpected keyword argument '%s'" % kwargs.pop()[0])
+        raise TypeError("xor() got an unexpected keyword argument '%s'" % kwargs.popitem()[0])
 
     if len(args) == 0:
         raise ValueError("Must have something to xor")

setup.py

@@ -59,7 +59,7 @@
 PythonH = os.path.join(get_python_inc(), 'Python.h')
 if not os.path.exists(PythonH):
     print("You must install the Python development headers!", file=sys.stderr)
-    print("$ apt-get install python-dev", file=sys.stderr)
+    print("$ sudo apt-get install python-dev", file=sys.stderr)
     sys.exit(-1)
 
 setup(