TypedDict help+completions, Any/Unvalidated dumping, scoped postponed annotations, abstract dataclass-like types and pydantic/attrs aliases

.github/workflows/tests.yaml

@@ -241,7 +241,7 @@ jobs:
           enable-cache: true
           cache-suffix: doctest
           cache-dependency-glob: pyproject.toml
-      - run: uv pip install -e .[all,shtab,doc]
+      - run: uv pip install -e .[all,test,shtab,doc]
       - name: Run doc tests
         run: sphinx-build -M doctest sphinx sphinx/_build sphinx/index.rst
 

CHANGELOG.rst

@@ -55,6 +55,25 @@ Added
   ``type[...]``, is now replaced by what it stands for: its PEP 696 ``default``,
   its constraints or its bound. Previously the value was accepted without any
   validation (`#953 <https://github.com/mauvilsa/jsonargparse/pull/953>`__).
+- Arguments typed as a ``TypedDict`` now have a ``--*.help`` option that shows
+  the keys that are accepted, their types and their descriptions. It receives no
+  value, unless the ``TypedDict`` is in a union with other types that have a
+  help, in which case the value is the name of the typed dict (`#956
+  <https://github.com/mauvilsa/jsonargparse/pull/956>`__).
+- ``add_class_arguments`` now accepts a ``TypedDict``, adding one argument per
+  key, analogous to a dataclass. ``instantiate`` gives the corresponding dict
+  (`#956 <https://github.com/mauvilsa/jsonargparse/pull/956>`__).
+- Descriptions of ``TypedDict`` keys taken from its docstring are now shown in
+  the help of ``**kwargs: Unpack[SomeTypedDict]`` parameters (`#956
+  <https://github.com/mauvilsa/jsonargparse/pull/956>`__).
+- ``shtab`` completion scripts now include the keys of a ``TypedDict``
+  argument, e.g. ``--data.key``, and the values that these keys accept (`#956
+  <https://github.com/mauvilsa/jsonargparse/pull/956>`__).
+- Pydantic's ``alias`` and ``validation_alias`` and attrs' ``alias`` are now
+  accepted as option and config keys, so that a parser accepts the same names as
+  the class itself. The parsed namespace and dumps use the name that the class
+  accepts, see :ref:`parameter-aliases` (`#956
+  <https://github.com/mauvilsa/jsonargparse/pull/956>`__).
 
 Fixed
 ^^^^^
@@ -84,11 +103,15 @@ Fixed
 - Signature parameters with a pydantic type nested in a container, e.g.
   ``list[HttpUrl]``, being skipped (`#948
   <https://github.com/mauvilsa/jsonargparse/pull/948>`__).
-- ``dump``, and thus ``--print_config``, failing when the value of an ``Any``
-  typed argument is a class instance that the config format can't represent. Now
-  these values are serialized as an import path, or as a message that says that
-  it was not serializable, see :ref:`unvalidated-types` (`#948
-  <https://github.com/mauvilsa/jsonargparse/pull/948>`__).
+- ``dump``, and thus ``--print_config``, failing when the value of an ``Any`` or
+  ``Unvalidated<...>`` typed argument is of a type that the config format does
+  not represent, e.g. a class instance, or a ``set`` when the format is json.
+  Now a type is derived from the value and used to serialize it, class instances
+  are serialized as an import path or as a message that says that it was not
+  serializable, and a warning is raised when the dumped value does not
+  round-trip, see :ref:`unvalidated-types` (`#948
+  <https://github.com/mauvilsa/jsonargparse/pull/948>`__, `#956
+  <https://github.com/mauvilsa/jsonargparse/pull/956>`__).
 - ``AssertionError`` without a message when adding an argument typed as a
   subscripted user defined generic class, e.g. ``Optional[Strategy[T]]`` (`#950
   <https://github.com/mauvilsa/jsonargparse/pull/950>`__).
@@ -173,6 +196,29 @@ Fixed
   <https://github.com/mauvilsa/jsonargparse/pull/954>`__).
 - Missing deprecation warning for the ``--print_config`` to ``--print_%s``
   change (`#955 <https://github.com/mauvilsa/jsonargparse/pull/955>`__).
+- ``shtab`` completions of ``**kwargs: Unpack[SomeTypedDict]`` parameters
+  showing ``NotRequired[...]`` as the expected type and not completing the
+  values of the keys that are not required (`#956
+  <https://github.com/mauvilsa/jsonargparse/pull/956>`__).
+- Postponed annotations of a method not resolving names that are defined in the
+  body of its class. Now the namespace of the class that defines the method is
+  used as locals (`#956 <https://github.com/mauvilsa/jsonargparse/pull/956>`__).
+- The ``class_path`` of an abstract class being accepted for a class typed
+  argument, only to fail on ``instantiate`` with ``TypeError: Can't instantiate
+  abstract class``. Now the parsing fails with ``Expected an instantiatable
+  class, but ... is abstract``, also when the ``class_path`` is implicit, i.e.
+  only init args given, and when the class is given by name (`#956
+  <https://github.com/mauvilsa/jsonargparse/pull/956>`__).
+- Values silently discarded on instantiation for pydantic fields that have an
+  alias and don't accept the attribute name, i.e. models and dataclasses without
+  ``populate_by_name``. Now the alias is the accepted name, see
+  :ref:`parameter-aliases` (`#956
+  <https://github.com/mauvilsa/jsonargparse/pull/956>`__).
+- attrs fields whose ``__init__`` parameter name differs from the attribute
+  name, i.e. an explicit ``alias`` or a private attribute, failing to
+  instantiate with ``TypeError: got an unexpected keyword argument`` or not
+  being configurable at all (`#956
+  <https://github.com/mauvilsa/jsonargparse/pull/956>`__).
 
 Changed
 ^^^^^^^
@@ -212,6 +258,12 @@ Changed
   option 'init_args....'``. Dataclass-like types now accept the same values
   whether or not they are added as a group, see :ref:`subclasses-disabled`
   (`#952 <https://github.com/mauvilsa/jsonargparse/pull/952>`__).
+- Dataclass-like types that are abstract, i.e. that have abstract methods or
+  inherit from ``abc.ABC``, now have subclass support enabled by default.
+  Previously such a type was unusable, since the ``class_path`` of an
+  implementation was rejected and giving its fields directly failed on
+  ``instantiate``. See :ref:`subclasses-disabled` (`#956
+  <https://github.com/mauvilsa/jsonargparse/pull/956>`__).
 
 Deprecated
 ^^^^^^^^^^

DOCUMENTATION.rst

@@ -515,7 +515,9 @@ Some notes about this support are:
 - Types that use components imported inside ``TYPE_CHECKING`` blocks are
   supported.
 
-- Resolving of forward references in types is supported.
+- Resolving of forward references in types is supported. This includes names
+  that are only defined in the body of the class that owns the method, e.g. a
+  nested class referred to without qualifying it.
 
 - Fully supported types are: ``str``, ``bool`` (more details in
   :ref:`boolean-arguments`), ``int``, ``float``, ``Decimal``, ``complex``,
@@ -531,12 +533,19 @@ Some notes about this support are:
 - ``dict``, ``Mapping``, ``MutableMapping``, ``MappingProxyType``,
   ``OrderedDict``, and ``TypedDict`` are supported but only with ``str`` or
   ``int`` keys. ``Required`` and ``NotRequired`` are also supported for
-  fine-grained specification of required/optional ``TypedDict`` keys.
-  ``Unpack`` is supported with ``TypedDict`` for more precise ``**kwargs``
-  typing as described in PEP `692 <https://peps.python.org/pep-0692/>`__.
-  For more details see :ref:`dict-items`. A ``TypedDict`` can also be used as
-  the argument of ``type``, e.g. ``type[SomeTypedDict]``, in which case the
-  value is an import path to a class. Since ``TypedDict`` classes don't support
+  fine-grained specification of required/optional ``TypedDict`` keys. ``Unpack``
+  is supported with ``TypedDict`` for more precise ``**kwargs`` typing as
+  described in PEP `692 <https://peps.python.org/pep-0692/>`__. For more details
+  see :ref:`dict-items`. The keys that a ``TypedDict`` argument accepts are
+  shown by a ``--*.help`` option, e.g. ``--data.help``. This option receives no
+  value, unless the ``TypedDict`` is in a union with other types that have their
+  own help, in which case the value is the name of the typed dict, e.g.
+  ``--data.help SomeTypedDict``. A ``TypedDict`` is also accepted by
+  :meth:`add_class_arguments <.ArgumentParser.add_class_arguments>`, which adds
+  one argument per key and on :meth:`instantiate <.ArgumentParser.instantiate>`
+  gives the corresponding dict. A ``TypedDict`` can also be used as the argument
+  of ``type``, e.g. ``type[SomeTypedDict]``, in which case the value is an
+  import path to a class. Since ``TypedDict`` classes don't support
   ``issubclass``, the given class is accepted when it is structurally
   compatible, as specified in PEP `589 <https://peps.python.org/pep-0589/>`__,
   i.e. it has all the keys of the expected ``TypedDict``, with the same types
@@ -726,13 +735,21 @@ must still be a list, though its items are not validated. Likewise, in a
 ``Union`` only the subtypes that can't be validated accept any value, the others
 are still validated as usual.
 
-Since there is no type to serialize with, a value of one of these parameters
-that a config format can't represent, e.g. a default that is an arbitrary
-object, is serialized in :meth:`dump <.ArgumentParser.dump>` and
-``--print_config`` the same as the instances given for a :ref:`subclass type
-<sub-classes>`. That is, as an import path when the value can be imported back,
-and otherwise as a message that says that it was not serializable, in which case
-a warning is also raised. The same applies to arguments typed as ``Any``.
+Since there is no type to serialize with, in :meth:`dump <.ArgumentParser.dump>`
+and ``--print_config`` a type is derived from the value itself, so that the
+value is serialized the same as it would be for an argument of that type. A
+value of a type that jsonargparse doesn't support, e.g. a default that is an
+arbitrary object, is serialized the same as the instances given for a
+:ref:`subclass type <sub-classes>`. That is, as an import path when the value
+can be imported back, and otherwise as a message that says that it was not
+serializable, in which case a warning is also raised.
+
+Parsing a dump back has no type to validate with either, so only the values that
+the config formats represent round-trip. For instance, a ``set`` is serialized
+as a list and parses back as a list, and an ``Enum`` member is serialized as its
+name and parses back as a string. A warning is raised for each dumped value that
+loses its type this way. All of the above equally applies to arguments typed as
+``Any``.
 
 
 .. _restricted-numbers:
@@ -2136,6 +2153,46 @@ the stubs. In these cases in the parser help the default is shown as
 ``Unknown<stubs-resolver>`` and not included in :meth:`get_defaults
 <.ArgumentParser.get_defaults>` or the output of ``--print_config``.
 
+.. _parameter-aliases:
+
+Parameter aliases
+^^^^^^^^^^^^^^^^^
+
+Pydantic and attrs allow giving a field a name that is different from the
+attribute name, an alias: pydantic's ``alias``/``validation_alias`` and attrs'
+``alias``. The resolvers take these aliases into account, so that a parser
+accepts the same names as the class itself.
+
+When the framework accepts both names, e.g. a pydantic model with
+``populate_by_name``, the alias is accepted as an additional option and config
+key. The attribute name is the one used in the parsed namespace, in
+``--print_config`` and in dumps:
+
+.. doctest:: parameter_aliases
+
+    >>> from pydantic import BaseModel, ConfigDict, Field
+
+    >>> class Client(BaseModel):
+    ...     model_config = ConfigDict(populate_by_name=True)
+    ...     api_key: str = Field(default="", alias="key")
+    ...
+
+    >>> parser = ArgumentParser()
+    >>> parser.add_class_arguments(Client, "client")  # doctest: +IGNORE_RESULT
+    >>> parser.parse_args(["--client.key=abc"])
+    Namespace(client=Namespace(api_key='abc'))
+
+When the framework only accepts the alias, e.g. the same model without
+``populate_by_name``, the alias is the name used everywhere, since giving the
+attribute name would not instantiate the class as expected.
+
+Aliases are not supported for a parameter whose type is a subclasses-disabled
+type added as a group of arguments, since then the name is a prefix of several
+arguments instead of a single option string. In this case only the attribute
+name is accepted. Enabling subclasses for the type, see
+:ref:`enable-disable-subclasses`, makes it a single argument, and then its alias
+is accepted as well.
+
 
 .. _dependency-injection:
 
@@ -2284,6 +2341,12 @@ be accepted. In this case the config would be like:
     type a class. The accepted ``init_args`` would be the parameters of that
     function.
 
+.. note::
+
+    Abstract classes, i.e. classes that have abstract methods, are not accepted
+    as ``class_path`` value, since they can't be instantiated. For the same
+    reason they are not included in the known subclasses shown in the help.
+
 
 .. _sub-config-files:
 
@@ -2715,6 +2778,11 @@ accepted values are the same. A subclass spec is accepted, though only with the
 "init_args": {"number": 8}}``. The ``class_path`` of a subclass is not accepted,
 unless subclass support is enabled for the type as described next.
 
+Abstract dataclass-like types are an exception. A class that has abstract
+methods or that inherits from ``abc.ABC`` is not intended to be instantiated
+from its own fields, so for these types subclass support is enabled by default,
+i.e. only the ``class_path`` of an implementation is accepted.
+
 
 .. _enable-disable-subclasses:
 
@@ -2736,7 +2804,8 @@ precedence over those in ``subclasses_disabled``. If a function name is given to
 ``subclasses_enabled``, it must correspond to a function previously registered
 in ``subclasses_disabled``; in this case, the effect is to unregister it. By
 default, the following disabling functions are registered: ``is_pure_dataclass``,
-``is_pydantic_model``, ``is_attrs_class``, and ``is_final_class``.
+``is_pydantic_model``, ``is_attrs_class``, and ``is_final_class``. These
+functions are not applied to abstract classes, see above.
 
 Some examples. Since ``subclasses_enabled`` takes precedence, it is possible to
 keep subclass support disabled for dataclasses, but enable it for a specific
@@ -3383,6 +3452,17 @@ giving as guidance which of the subclasses accepts it. An example would be:
     $ example.py --cls other.module.SubclassA --cls.param2 <TAB><TAB>
     Expected type: int; Accepted by subclasses: SubclassA
 
+Analogously, for dataclass-like types and ``TypedDict``, the fields or keys are
+completed, as well as the values that they accept, e.g.:
+
+.. code-block:: bash
+
+    $ example.py --data.<TAB><TAB>
+    --data.verbose    --data.mode
+    $ example.py --data.verbose <TAB><TAB>
+    Expected type: bool; 2/2 matched choices
+    true  false
+
 argcomplete
 -----------
 

jsonargparse/_actions.py

@@ -334,12 +334,6 @@
 class _ActionHelpClassPath(NonParsingAction):
     sub_add_kwargs: dict[str, Any] = {}
 
-    @classmethod
-    def get_help_types(cls, typehint) -> tuple | None:
-        from ._typehints import get_subclass_or_closed_types
-
-        return get_subclass_or_closed_types(typehint=typehint, also_lists=True, callable_return=True)
-
     def __init__(self, typehint=None, **kwargs):
         if typehint is not None:
             self._typehint = typehint
@@ -347,25 +341,34 @@
             self.update_init_kwargs(kwargs)
             super().__init__(**kwargs)
 
     def update_init_kwargs(self, kwargs):
-        from ._typehints import is_protocol
+        from ._typehints import get_help_types, is_protocol, is_typed_dict
 
         self._typehint = kwargs.pop("_typehint")
-        self._help_types = self.get_help_types(self._typehint)
+        self._help_types = get_help_types(self._typehint)
         assert self._help_types and all(isinstance(b, type) for b in self._help_types)
-        self._single_class = len(self._help_types) == 1 and is_subclasses_disabled(self._help_types[0])
+        typed_dicts = [t for t in self._help_types if is_typed_dict(t)]
+        # a single type means that the help refers to it, so no value is expected
+        single_type = len(self._help_types) == 1 and (is_subclasses_disabled(self._help_types[0]) or bool(typed_dicts))
         self._basename = iter_to_set_str(t.__name__ for t in self._help_types)
 
         if len(self._help_types) == 1:
-            kwargs["nargs"] = 0 if self._single_class else "?"
+            kwargs["nargs"] = 0 if single_type else "?"
 
-        if self._single_class:
+        if single_type:
             msg = ""
         else:
             kwargs["metavar"] = "CLASS_PATH_OR_NAME"
             self._kind = "subclass of"
             if any(is_protocol(b) for b in self._help_types):
                 self._kind = "subclass or implementer of protocol"
+            if typed_dicts:
+                # a typed dict is given by name, since it doesn't accept a class path
+                if len(typed_dicts) == len(self._help_types):
+                    kwargs["metavar"] = "NAME"
+                    self._kind = "typed dict"
+                else:
+                    self._kind = "class or typed dict"
             msg = f"the given {self._kind} "
 
         kwargs["default"] = SUPPRESS
@@ -377,24 +380,31 @@
             return type(self)(**kwargs)
         return self.print_help(args)
 
+    def resolve_help_type(self, value, option_string):
+        from ._typehints import implements_protocol, is_typed_dict, resolve_class_path_by_name
+
+        if self.nargs == 0 or (self.nargs == "?" and value is None):
+            return self._help_types[0]
+        typed_dict = next((t for t in self._help_types if is_typed_dict(t) and t.__name__ == value), None)
+        if typed_dict:
+            return typed_dict
+        # typed dicts excluded since they don't have subclasses that a class path could refer to
+        class_types = tuple(t for t in self._help_types if not is_typed_dict(t))
+        val_class = None
+        if class_types:
+            try:
+                val_class = import_object(resolve_class_path_by_name(class_types, value))
+            except Exception as ex:
+                raise TypeError(f"{option_string}: {ex}") from ex
+        if not any(is_subclass(val_class, b) or implements_protocol(val_class, b) for b in class_types):
+            raise TypeError(f'{option_string}: "{value}" is not a {self._kind} {self._basename}')
+        return val_class
+
     def print_help(self, call_args):
-        from ._typehints import (
-            adapt_partial_callable_class,
-            implements_protocol,
-            resolve_class_path_by_name,
-        )
+        from ._typehints import adapt_partial_callable_class
 
         parser, _, value, option_string = call_args
-        try:
-            if self.nargs == 0 or (self.nargs == "?" and value is None):
-                val_class = self._help_types[0]
-            else:
-                val_class = import_object(resolve_class_path_by_name(self._help_types, value))
-        except Exception as ex:
-            raise TypeError(f"{option_string}: {ex}") from ex
-
-        if not any(is_subclass(val_class, b) or implements_protocol(val_class, b) for b in self._help_types):
-            raise TypeError(f'{option_string}: Class "{value}" is not a {self._kind} {self._basename}')
+        val_class = self.resolve_help_type(value, option_string)
         dest = re.sub("\\.help$", "", self.dest)
         subparser = type(parser)(description=f"Help for {option_string}={get_import_path(val_class)}")
         val = Namespace(class_path=get_import_path(val_class))