proposal: fmt: recognize and use StringAppend method to avoid required allocation in String

[soypat]

Context

Heap allocations are almost unavoidable in Go. Most of the language has been designed around having an excellent garbage collector. That said, there is a growing interest in being more mindful of how we allocate memory. See proposals #51317 (arenas) #71497 (json/v2). The embedded Go community is also gaining traction as powerful compute becomes more available on small devices such as the recent RP2350 MCU, which was quickly supported by TinyGo due to widespread community driven effort by new contributors.

One of the more common methods of allocating memory in Go is through the fmt.Stringer interface which permeates a big part of the Go ecosystem. Most of the time it is invisible, used from inside fmt.Print* family of functions. This method serves a very important purpose in Go which is to give a text visual of a data structure or concept abstracted by a data structure.

The issue with this interface is that when the string is not constant and must be built it allocates, always. This presents an issue for developers seeking to minimize or constrain allocations. For sure if some of these developers were given a fmt.Stringer interface which constrains allocations they would prefer it to the current fmt.Stringer interface.

Proposal

I propose we add a fmt.StringAppender interface with the following signature:

type StringAppender interface {
    StringAppend([]byte) []byte
}

The method would do what one would expect, append the string representation to the buffer and return the extended result, just like the result of fmt.Stringer.

Users could implement the fmt.String* method set by first implementing StringAppend which would define the result of fmt.Stringer by simply calling the StringAppend method on a nil byte slice: string(Hexa(1).StringAppend(nil)).

type Hexa int

func (h Hexa) StringAppend(b []byte) []byte {
	return fmt.Appendf(b, "0x%x", h)
}

[soypat]

Is this considered a duplicate of AppendText for encoding?

[seankhliao]

yes

[soypat]

It would seem they fulfill different purposes. String method is meant to return a "native" format which need not necessarily be a format that can be decoded back, hence the lack of error in the function signature.

[ianlancetaylor]

For cases where the fact that the fmt package calls a String method causes excessive heap allocation, it is already possible to implement a Format method.

[soypat]

I would argue implementing a Format method is much harder and also would require the importing of the fmt package, which is not something one just does on embedded projects.

Calling Write with a byte buffer also allocates unless the buffer is owned by the calling data structure, which also unecessarily complicates what you are trying to do. This also requires writing the data twice, once to format it and another to write it to the State.

I don't think fmt.Formatter is a solution to the issue. It serves a different purpose.

[ianlancetaylor]

I am concerned that String methods are extremely common. Adding a new StringAppend method that many types are expected to implement is going to cause a vast ecosystem effect, a cost that seems likely to exceed the benefit.

However, I agree that this is not a duplicate of AppendText, so I'll reopen.

[seankhliao]

I'd be concerned this adds more complexity for little gain. Even the existing TextAppender has quite low adoption compared to TextMarshaler

TextAppender 148 hits: https://github.com/search?q=language%3AGo+%2F%5Efunc+%5C%28.*%5C%29+AppendText%5C%28.*byte%5C%29.*byte%2F&type=code
TextMarshaler 28.8k hits: https://github.com/search?q=language%3AGo+%2F%5Efunc+%5C%28.*%5C%29+MarshalText%5C%28%5C%29.*byte%2F&type=code

Also, what would be expected to check for and use this interface? Since you mentioned not importing fmt, it doesn't seem like anything in the standard library would use this?

[gabyhelp]

Related Issues

• fmt: add Append, Appendf, Appendln #47579 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[ianlancetaylor]

I think the idea is that the type generating the output would not import fmt. However, the code that does the printing would be using fmt.Printf and friends as usual.

I agree that if the program as a whole is going to import fmt, it does not seem like a hardship for the type generating the output to import fmt.

[soypat]

fmt.Append* family of methods are the most obvious candidates to check for the method for users who do import fmt. Users who don't import fmt would probably use the raw StringAppend method on a type to marshal the data onto preallocated buffers.

It'd be nice to have adoption in the standard library among common types such as time.Duration which see a lot of use when writing embedded programs. As the interface becomes more widespread hopefully those mindful of heap allocations would start adding this method to their types.

[seankhliao]

I don't quite understand the objection to using TextMarshaler / TextAppender. Both represent textual representations of the value, and it seems a significant portion of code is happy to share the output for String / MarshalText.
The existence of MarshalText doesn't mean that a corresponding UnmarshalText has to exist, that's why they're separate interfaces.

using String for MarshalText: https://github.com/search?q=language%3AGo+%2F%5Efunc+%5C%28.*%5C%29+MarshalText%5C%28%5C%29.*byte.*%5Cn.*%5C.String%2F&type=code
using MarshalText for String: https://github.com/search?q=language%3AGo+%2F%5Efunc+%5C%28.*%5C%29+String%5C%28%5C%29+string+%5C%7B%5Cn.*MarshalText%2F&type=code

[soypat]

People are using the result of MarshalText and String interchangeably. Does this mean we can override %s formatting verb in a call to fmt.Appendf with the result of AppendText? Would the result of time.Second.AppendText be 1s, just like the String method?

If you want to argue that way we should be ready to standardize the encoding.TextAppender and encoding.TextMarshaler as having the same result as the fmt.Stringer, in which case I'm not sure why we even thought of different packages for them in the first place.