-
-
Notifications
You must be signed in to change notification settings - Fork 2.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[PEP 747] Recognize TypeForm[T] type and values (#9773) #18690
base: master
Are you sure you want to change the base?
Conversation
User must opt-in to use TypeForm with --enable-incomplete-feature=TypeForm In particular: * Recognize TypeForm[T] as a kind of type that can be used in a type expression * Recognize a type expression literal as a TypeForm value in: - assignments - function calls - return statements * Define the following relationships between TypeForm values: - is_subtype - join_types - meet_types * Recognize the TypeForm(...) expression * Alter isinstance(typx, type) to narrow TypeForm[T] to Type[T]
d74b387
to
ca4c79f
Compare
for more information, see https://pre-commit.ci
There are many unexpected test suite errors when run in CI. I'll take a look in the next few days. |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
What are the reasons for this? Maybe we can adjust something to make it possible to analyze type expressions in TypeChecker.
Is this specified in the PEP? If they should be allowed in other contexts, we should try to make it possible. For example, if we could parse type expressions in TypeChecker, we could perhaps use the type context to decide whether something might be a TypeForm. The idea would be like this (haven't thought about this very carefully though):
This will increase memory use (on average 8 bytes per expression node in compiled mypy). Again, I'm in favor of finding a way to avoid this. |
And thanks for working on this! This will be a useful feature, and will enable many new use cases. |
This comment has been minimized.
This comment has been minimized.
…mypyc Recognize the remaining subtypes of MaybeTypeExpression.
…mypyc Fix: Workaround mypy thinking incorrectly that isinstance(X, Union[...]) does not work at runtime.
Diff from mypy_primer, showing the effect of this PR on open source code: Tanjun (https://github.com/FasterSpeeding/Tanjun)
beartype (https://github.com/beartype/beartype)
+ beartype/_util/hint/nonpep/api/utilmodpandera.py:178: error: Argument "hint" to "get_hint_pep484585_generic_base_in_module_first" has incompatible type "object"; expected "TypeForm[Any]" [arg-type]
+ beartype/_data/hint/datahintpep.py:93: error: Unused "type: ignore" comment [unused-ignore]
+ beartype/_data/hint/datahintpep.py:98: error: TypeForm is experimental, must be enabled with --enable-incomplete-feature=TypeForm [misc]
- beartype/_util/hint/pep/proposal/pep484585/pep484585container.py:68: error: Unused "type: ignore" comment [unused-ignore]
+ beartype/_util/hint/pep/proposal/pep585.py:204: error: Argument 1 to "is_hint_pep585_generic_unsubscripted" has incompatible type "TypeForm[Any] | None"; expected "TypeForm[Any]" [arg-type]
+ beartype/_util/hint/pep/proposal/pep585.py:370: error: Incompatible types in assignment (expression has type "type | None", variable has type "TypeForm[Any]") [assignment]
+ beartype/_util/hint/pep/proposal/pep484585/pep484585container.py:72: error: Incompatible types in assignment (expression has type "<typing special form>", variable has type "TypeForm[Any]") [assignment]
+ beartype/_util/hint/pep/proposal/pep484585/pep484585.py:61: error: Incompatible return value type (got "TypeForm[Any] | tuple[TypeForm[Any], ...]", expected "TypeForm[Any]") [return-value]
+ beartype/_util/hint/pep/proposal/pep484/pep484union.py:69: error: Incompatible return value type (got "object", expected "TypeForm[Any]") [return-value]
+ beartype/_util/hint/pep/proposal/pep484/pep484generic.py:66: error: Argument 1 to "is_hint_pep484_generic_unsubscripted" has incompatible type "TypeForm[Any] | None"; expected "TypeForm[Any]" [arg-type]
- beartype/_check/metadata/hint/hintmeta.py:159: error: Unused "type: ignore" comment [unused-ignore]
- beartype/_util/hint/pep/proposal/pep484585/pep484585func.py:106: error: Unused "type: ignore" comment [unused-ignore]
+ beartype/_util/hint/pep/proposal/pep484585/generic/pep484585genget.py:280: error: List item 0 has incompatible type "list[TypeForm[Any] | None]"; expected "list[TypeForm[Any] | list[TypeForm[Any]] | _HintBaseData]" [list-item]
+ beartype/_util/hint/pep/proposal/pep484585/generic/pep484585genget.py:354: error: Argument 1 to "is_hint_pep484585_generic_user" has incompatible type "TypeForm[Any] | list[TypeForm[Any]] | _HintBaseData"; expected "TypeForm[Any]" [arg-type]
+ beartype/_util/hint/pep/proposal/pep484585/generic/pep484585genget.py:375: error: Argument "hint" to "get_hint_pep484585_generic_bases_unerased" has incompatible type "TypeForm[Any] | list[TypeForm[Any]] | _HintBaseData"; expected "TypeForm[Any]" [arg-type]
+ beartype/_util/hint/pep/proposal/pep484585/generic/pep484585genget.py:431: error: Incompatible types in assignment (expression has type "list[TypeForm[Any]] | TypeForm[Any] | _HintBaseData", variable has type "list[TypeForm[Any]]") [assignment]
+ beartype/_util/hint/pep/proposal/pep484585/generic/pep484585genget.py:499: error: Invalid index type "TypeForm[Any]" for "dict[TypeVar, TypeForm[Any]]"; expected type "TypeVar" [index]
- beartype/_util/hint/pep/proposal/pep484585/generic/pep484585genget.py:714: error: Unused "type: ignore" comment [unused-ignore]
+ beartype/_util/hint/pep/proposal/pep484585/generic/pep484585genget.py:522: error: Incompatible types in assignment (expression has type "Any | TypeForm[Any] | list[TypeForm[Any]] | _HintBaseData", target has type "TypeForm[Any]") [assignment]
+ beartype/_util/hint/pep/proposal/pep484585/generic/pep484585genget.py:553: error: Invalid index type "TypeForm[Any]" for "dict[TypeVar, TypeForm[Any]]"; expected type "TypeVar" [index]
+ beartype/_util/hint/pep/proposal/pep484585/generic/pep484585genget.py:554: error: Incompatible types in assignment (expression has type "Any | TypeForm[Any] | list[TypeForm[Any]] | _HintBaseData", target has type "TypeForm[Any]") [assignment]
+ beartype/_util/hint/pep/proposal/pep484585/generic/pep484585genget.py:589: error: Argument "hint" to "get_hint_pep484585_generic_type" has incompatible type "TypeForm[Any] | list[TypeForm[Any]] | _HintBaseData"; expected "TypeForm[Any]" [arg-type]
+ beartype/_util/hint/pep/proposal/pep484585/generic/pep484585genget.py:689: error: No overload variant of "get" of "dict" matches argument types "TypeForm[Any]", "TypeForm[Any]" [call-overload]
+ beartype/_util/hint/pep/proposal/pep484585/generic/pep484585genget.py:689: note: Possible overload variants:
+ beartype/_util/hint/pep/proposal/pep484585/generic/pep484585genget.py:689: note: def get(self, TypeVar, /) -> TypeForm[Any] | None
+ beartype/_util/hint/pep/proposal/pep484585/generic/pep484585genget.py:689: note: def get(self, TypeVar, TypeForm[Any], /) -> TypeForm[Any]
+ beartype/_util/hint/pep/proposal/pep484585/generic/pep484585genget.py:689: note: def [_T] get(self, TypeVar, _T, /) -> TypeForm[Any] | _T
- beartype/_util/hint/pep/utilpepget.py:1088: error: Unused "type: ignore" comment [unused-ignore]
+ beartype/_check/convert/_reduce/_pep/redpep544.py:79: error: Incompatible return value type (got "<typing special form>", expected "TypeForm[Any]") [return-value]
+ beartype/_util/hint/pep/utilpepget.py:1219: error: Incompatible return value type (got "TypeForm[Any] | None", expected "type | None") [return-value]
+ beartype/_check/convert/_reduce/_pep/redpep695.py:59: error: Incompatible types in assignment (expression has type "None", variable has type "TypeForm[Any]") [assignment]
+ beartype/_check/forward/fwdmain.py:245: error: Incompatible return value type (got "str", expected "TypeForm[Any]") [return-value]
+ beartype/_check/convert/_reduce/_pep/redpep484604.py:138: error: Incompatible types in assignment (expression has type "<typing special form>", variable has type "TypeForm[Any]") [assignment]
+ beartype/_check/convert/_reduce/_pep/pep484585/redpep484585generic.py:95: error: Incompatible return value type (got "<typing special form>", expected "TypeForm[Any] | HintSanifiedData") [return-value]
+ beartype/_check/convert/_reduce/_pep/pep484585/redpep484585generic.py:129: error: Incompatible types in assignment (expression has type "TypeForm[Any] | HintSanifiedData", variable has type "TypeForm[Any]") [assignment]
+ beartype/_check/convert/_reduce/_pep/pep484/redpep484typevar.py:96: error: No overload variant of "get" of "dict" matches argument types "TypeForm[Any]", "TypeForm[Any]" [call-overload]
+ beartype/_check/convert/_reduce/_pep/pep484/redpep484typevar.py:96: note: Possible overload variants:
+ beartype/_check/convert/_reduce/_pep/pep484/redpep484typevar.py:96: note: def get(self, TypeVar, /) -> TypeForm[Any] | None
+ beartype/_check/convert/_reduce/_pep/pep484/redpep484typevar.py:96: note: def get(self, TypeVar, TypeForm[Any], /) -> TypeForm[Any]
+ beartype/_check/convert/_reduce/_pep/pep484/redpep484typevar.py:96: note: def [_T] get(self, TypeVar, _T, /) -> TypeForm[Any] | _T
+ beartype/_check/convert/_reduce/_nonpep/api/redapinumpy.py:243: error: Incompatible return value type (got "<typing special form>", expected "TypeForm[Any]") [return-value]
+ beartype/door/_cls/doormeta.py:157: error: Argument 1 to "is_hint_cacheworthy" has incompatible type "object"; expected "TypeForm[Any]" [arg-type]
+ beartype/door/_cls/doormeta.py:248: error: Argument 1 to "get_typehint_subclass" has incompatible type "object"; expected "TypeForm[Any]" [arg-type]
+ beartype/_check/convert/_convcoerce.py:190: error: Incompatible return value type (got "<typing special form>", expected "TypeForm[Any]") [return-value]
+ beartype/_check/convert/_convcoerce.py:408: error: Incompatible return value type (got "object", expected "TypeForm[Any]") [return-value]
+ beartype/_check/convert/_reduce/redhint.py:252: error: Incompatible types in assignment (expression has type "Iota", variable has type "TypeForm[Any] | HintSanifiedData") [assignment]
+ beartype/_check/convert/_reduce/redhint.py:297: error: Incompatible return value type (got "<typing special form>", expected "TypeForm[Any] | HintSanifiedData") [return-value]
+ beartype/_check/convert/_reduce/redhint.py:372: error: Incompatible return value type (got "<typing special form>", expected "TypeForm[Any] | HintSanifiedData") [return-value]
+ beartype/_check/convert/_reduce/redhint.py:387: error: Incompatible return value type (got "<typing special form>", expected "TypeForm[Any] | HintSanifiedData") [return-value]
+ beartype/_check/error/errcause.py:265: error: Incompatible default for argument "hint" (default has type "Iota", argument has type "TypeForm[Any]") [assignment]
+ beartype/_check/error/errcause.py:265: note: Error code "assignment" not covered by "type: ignore" comment
- beartype/_check/error/_errtype.py:91: error: Unused "type: ignore" comment [unused-ignore]
- beartype/_check/error/_errtype.py:202: error: Unused "type: ignore" comment [unused-ignore]
- beartype/_check/error/_errtype.py:269: error: Unused "type: ignore" comment [unused-ignore]
- beartype/_check/code/codemake.py:542: error: Unused "type: ignore" comment [unused-ignore]
- beartype/_check/code/codemake.py:991: error: Unused "type: ignore" comment [unused-ignore]
+ beartype/_check/error/errcause.py:266: error: Incompatible default for argument "hint_or_sane" (default has type "Iota", argument has type "TypeForm[Any] | HintSanifiedData") [assignment]
+ beartype/_check/error/errcause.py:266: note: Error code "assignment" not covered by "type: ignore" comment
+ beartype/_check/error/errcause.py:405: error: Incompatible types in assignment (expression has type "None", variable has type "TypeForm[Any]") [assignment]
+ beartype/_check/error/errcause.py:411: error: Incompatible types in assignment (expression has type "None", variable has type "TypeForm[Any] | HintSanifiedData") [assignment]
+ beartype/_check/code/_pep/codepep484604.py:187: error: Argument 1 to "add" of "set" has incompatible type "TypeForm[Any]"; expected "type" [arg-type]
+ beartype/_check/error/errget.py:235: error: Argument "hint" to "get_hint_object_violation" has incompatible type "TypeForm[Any] | Iota"; expected "TypeForm[Any]" [arg-type]
+ beartype/_check/error/_errtype.py:328: error: Incompatible types in assignment (expression has type "TypeForm[Any]", variable has type "type | tuple[type, ...]") [assignment]
+ beartype/_check/code/codemake.py:261: error: Incompatible types in assignment (expression has type "None", variable has type "TypeForm[Any]") [assignment]
+ beartype/_check/code/codemake.py:269: error: Incompatible types in assignment (expression has type "None", variable has type "TypeForm[Any]") [assignment]
+ beartype/_check/code/codemake.py:278: error: Incompatible types in assignment (expression has type "None", variable has type "TypeForm[Any] | HintSanifiedData") [assignment]
- beartype/_check/code/codemake.py:1363: error: Unused "type: ignore" comment [unused-ignore]
+ beartype/_check/code/codemake.py:1377: error: Incompatible types in assignment (expression has type "tuple[Any, ...]", variable has type "TypeForm[Any]") [assignment]
- beartype/_check/code/codemake.py:1389: error: Unused "type: ignore" comment [unused-ignore]
+ beartype/_check/code/codemake.py:1400: error: Argument 1 to "add_func_scope_type_or_types" of "HintsMeta" has incompatible type "TypeForm[Any]"; expected "type | tuple[type, ...] | AbstractSet[type]" [arg-type]
+ beartype/_check/code/codemake.py:1493: error: Argument 1 to "add_func_scope_type_or_types" of "HintsMeta" has incompatible type "TypeForm[Any]"; expected "type | tuple[type, ...] | AbstractSet[type]" [arg-type]
+ beartype/door/_func/doorcheck.py:133: error: Argument 1 to "make_func_raiser" has incompatible type "object"; expected "TypeForm[Any]" [arg-type]
+ beartype/_decor/wrap/_wrapreturn.py:92: error: Incompatible types in assignment (expression has type "object", variable has type "TypeForm[Any]") [assignment]
+ beartype/_decor/wrap/_wrapargs.py:178: error: Incompatible types in assignment (expression has type "None", variable has type "TypeForm[Any]") [assignment]
+ beartype/_decor/wrap/_wrapargs.py:183: error: Incompatible types in assignment (expression has type "None", variable has type "TypeForm[Any] | HintSanifiedData") [assignment]
+ beartype/_decor/wrap/_wrapargs.py:244: error: Incompatible types in assignment (expression has type "object", variable has type "TypeForm[Any]") [assignment]
+ beartype/door/_cls/doorsuper.py:519: error: Argument 1 to "sanify_hint_child" has incompatible type "T"; expected "TypeForm[Any]" [arg-type]
- beartype/door/_cls/pep/pep484/doorpep484newtype.py:70: error: Unused "type: ignore" comment [unused-ignore]
|
Analyzing an The logic of I agree that it would be ideal if TypeAnalyser could be used directly inside TypeChecker. It would eliminate several of the controversial issues mentioned at the top of this PR.
Agreed. The PEP doesn't limit the syntactic locations where type form literals should be recognized, so the current implementation which only recognizes in certain locations isn't ideal.
The latest version of this PR avoids the issue by moving the "as_type" attribute out of Again, if TypeChecker could parse a type expression itself there would be no need to store a parsed |
I'm no longer sure if this is easy to do, since string literal types wouldn't be semantically analyzed, and so we'd need to parse and analyze them from scratch during type checking, and doing this could be tricky. Your current approach may be better after all. To make my alternative approach work with string literal types would imply calling the semantic analyzer from the type checker to analyze AST subtrees that are used in a type form context. I'm not sure if this is practical to do, but it might be. If we can do this, then we could probably use the existing type analyzer as well, so the implementation would be simpler. This could be the best approach (if practical):
I'll think about this more. |
Here is some more brainstorming, mainly about how to make the current approach more efficient.
So in summary, a mix of your current approach and my proposed approach could be the winning formula:
I may have missed some issues, but hopefully this gives some useful ideas. |
I investigated whether a TypeAnalyzer could be created inside the TypeChecker pass. It looks like it might be possible:
It's not clear to me that the behavior of SemanticAnalyzer's lookup methods are aligned enough with TypeChecker.lookup_qualified() to make it reasonable to implement the former on top of the latter. @JukkaL, any insights on this question? |
Implements the TypeForm PEP 747, as an opt-in feature enabled by the CLI flag
--enable-incomplete-feature=TypeForm
.Implementation approach:
The
TypeForm[T]
is represented as a type using the existingTypeType
class, with anis_type_form=True
constructor parameter.Type[C]
continues to be represented usingTypeType
, but withis_type_form=False
(the default).Recognizing a type expression literal such as
int | str
requires parsing anExpression
as a type expression. Only the SemanticAnalyzer pass appears to have the ability to parse type expressions, usingSemanticAnalyzer.expr_to_analyzed_type()
. In particular the laterTypeChecker
pass does not appear to have the ability to parse type expressions.Therefore during the SemanticAnalyzer pass, at certain syntactic locations (i.e. assignment r-values, callable arguments, returned expressions), the analyzer tries to parse the
Expression
it is looking at usingtry_parse_as_type_expression()
- a new function - and stores the result (aType
) inExpression.as_type
- a new attribute.During the later TypeChecker pass, when looking at an
Expression
to determine its type, if the expression is in a type context that expects some kind ofTypeForm[...]
and the expression was successfully parsed as a type expression by the earlier SemanticAnalyzer pass, the expression will be given the typeTypeForm[expr.as_type]
rather than using the regular type inference rules for a value expression.Key relationships between
TypeForm[T]
,Type[C]
, andobject
types are defined in the visitors poweringis_subtype
,join_types
, andmeet_types
.The
TypeForm(T)
expression is recognized as aTypeFormExpr
and has the return typeTypeForm[T]
.The new test suite in
check-typeform.test
is a good reference to the expected behaviors for operations that interact withTypeForm
in some way.Controversial parts of this PR, in @davidfstr 's opinion:
Type form literals are only recognized in certain syntactic locations (and not ALL possible locations). Namely they are recognized as (1) assignment r-values, (2) callable expression arguments, and (3) as returned expressions, but nowhere else. For example they aren't recognized in expressions like
dict_with_typx_keys[int | str]
.SemanticAnalyzer makes very many unnecessary calls to try_parse_as_type_expression() - a call for every syntactic location where recognizing a type form literal is supported, potentially doubling the type to parse value expressions. It would be much preferred if
try_parse_as_type_expression()
could only be called when TypeChecker is looking at an expression known to be in aTypeForm[T]
type context. Unfortunately TypeChecker cannot parse type expressions itself; only the earlier SemanticAnalyzer pass can do that.try_parse_as_type_expression()
saves and restores a lot of SemanticAnalyzer state, so that failing to parse a type expression literal doesn't cause unexpected side effects.try_parse_as_type_expression()
.The(Edit: Fixed.)Expression
class no longer declares (empty)__slots__
and now always includes anas_type
attributeI wonder if this will increase memory usage forExpression
objects enough to noticeably decrease mypy performance. It's unclear to me what tools are available to check for such an issue.The existing
TypeType
class is now used to represent BOTH theType[T]
andTypeForm[T]
types, rather than introducing a distinct subclass ofType
to represent theTypeForm[T]
type. This was done to simplify logic that manipulates bothType[T]
andTypeForm[T]
values, since they are both manipulated in very similar ways.The "normalized" form of
TypeForm[X | Y]
- as returned byTypeType.make_normalized()
- is justTypeForm[X | Y]
rather thanTypeForm[X] | TypeForm[Y]
, differing from the normalization behavior ofType[X | Y]
.