Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ e este projeto adere ao [Versionamento Semântico](https://semver.org/lang/pt-BR
- Nos providers `claude_code` e `codex`, o status HTTP 408 ou 409 dá `ProviderTransientError`, e a linha é repetida com backoff, como no erro de SDK que chega pelo LangChain; no 408, o aviso sai com a caixa de timeout. Antes, os dois status davam `ProviderError` definitivo, e a linha ficava com erro na primeira falha (#274).
- A caixa genérica de erro (`ERRO NO PROCESSAMENTO`) mostra a mensagem em até três linhas, terminada em reticências quando passa disso, e a moldura fica alinhada nas linhas do tipo e dos detalhes. Antes, a mensagem era cortada em 66 caracteres sem aviso, e no `claude_code` o corte caía dentro do prefixo `Claude Code SDK retornou erro (subtipo, status N): `, sem mostrar nada do texto do SDK (#275).
- No provider `claude_code`, o `api_error_status` que o SDK entrega como texto de dígitos ou float inteiro, como `"401"` ou `401.0`, é classificado como o status inteiro: `"401"` dá a caixa de autenticação sem nova tentativa, e `"429"` reduz o paralelismo. `True`, `False` e os demais valores contam como status ausente; antes, `True` passava como status e virava `ProviderError` definitivo com `status_code=True` (#276).
- O campo `deque`, de `collections` ou `typing.Deque`, passa a contar como estrutura em `get_complex_fields`: o checkpoint o grava em JSON, também dentro de outra estrutura, como `dict[str, deque[str]]`, e a retomada o devolve como `deque`. Antes, a retomada do CSV e do XLSX recusava a linha concluída e pedia `reprocess_columns`, e a do parquet, cuja gravação falhava, pagava de novo as linhas concluídas. A estrutura passa a ser reconhecida pelas ABCs `Sequence`, `Set` e `Mapping`, menos texto e bytes (#287).

## [0.11.1] - 2026-10-29

Expand Down
2 changes: 1 addition & 1 deletion docs/en/guides/performance.md
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,7 @@ result = dataframeit(
)
```

The format is inferred from the file extension (`.csv`, `.xlsx`, `.parquet`). In every format, the model's structure fields, that is, list, tuple, set, dict, `Sequence`, `AbstractSet`, `Mapping`, `TypedDict`, `NamedTuple` and nested model, are saved as JSON text, sets in a fixed order, and `Enum` values by their value; `read_df` turns the structures back, and resuming returns structures and date, time, `datetime`, `Enum`, `int` and `bool` fields with their declared type, also with `strict=True`: a date saved as text, an integer read back as a float and a tuple read back as a list come back as `date`, `int` and `tuple`. CSV and XLSX write an empty string and `None` as the same empty cell. When resuming, that cell comes back as `""` in a text field that does not accept `None`, such as `str` or `str = "default"`, and as `None` in a field that accepts it, such as `Optional[str]`; to keep the `""` of an optional field, use parquet. In a `str` field whose default is `None`, given directly or by a factory, it also comes back as `None`. The empty string of a field whose factory reads the other fields is not restored: the empty cell counts as missing, and resuming fills it with the factory's value, even where the run saved `""`. A validator that turns an answer into `None` in a `str` field gives a value that the field's type rejects. In parquet, resuming flags that `None` for reprocessing. In CSV and XLSX, it flags the field when the validator turns the empty string itself into `None`, and returns `""` when it turns another answer, such as `"N/A"`, into `None`, because the empty cell does not say which of the two was saved. To keep that `None`, declare the field as `Optional[str]`. This holds for a run that applies the model's validators; in per-field mode (`search_per_field`), they do not run during execution, which saves the empty string as it came, and resuming from CSV and XLSX flags the field whose validator turns the empty string into `None`, while resuming from parquet returns the saved `""`. Parquet stores the empty string and `None` apart, and resuming returns the empty string as it was saved. In XLSX, text that starts with `=` is saved as text, not as a formula. XLSX stores numbers as doubles, and an integer above 2**53 comes back rounded; for ids that large, use CSV or parquet. None of the three formats keeps the NaN of a `float` field apart from null: CSV and XLSX write both as an empty cell, and parquet as null. When resuming, the NaN comes back missing, and a required field that received it is flagged for reprocessing. In parquet, an integer above 2**63 - 1 next to a pending row and an integer above 2**53 next to a float in the same column are not saved, and the checkpoint save fails with a warning. If execution is interrupted, run the same call again with the same input: with `resume=True`, the default, and the file present, execution continues from the checkpoint. On every save, `<checkpoint>.dataframeit.json` is written next to it with the run's signature (prompt, provider, model, `model_kwargs` except `timeout`, schema, search, and the text and status columns), a hash of each row's text, and a hash of the checkpoint itself. The checkpoint is resumed only when all three match; with another configuration, another input, or a file that is not the one the signature describes, the run warns, starts over, and overwrites it. In a row already concluded, successfully or with an error, the model fields come from the checkpoint, over whatever the input carried in them. To keep an output corrected by hand as it is, pass it with the status column, and the checkpoint is not read again. A complete output is returned without it, and the column comes back with `output["_dataframeit_status"] = "processed"` (or the name passed in `status_column`). With `resume=False`, the file is ignored and overwritten.
The format is inferred from the file extension (`.csv`, `.xlsx`, `.parquet`). In every format, the model's structure fields, that is, list, tuple, `deque`, set, dict, `Sequence`, `AbstractSet`, `Mapping`, `TypedDict`, `NamedTuple` and nested model, are saved as JSON text, sets in a fixed order, and `Enum` values by their value; `read_df` turns the structures back, and resuming returns structures and date, time, `datetime`, `Enum`, `int` and `bool` fields with their declared type, also with `strict=True`: a date saved as text, an integer read back as a float and a tuple read back as a list come back as `date`, `int` and `tuple`. CSV and XLSX write an empty string and `None` as the same empty cell. When resuming, that cell comes back as `""` in a text field that does not accept `None`, such as `str` or `str = "default"`, and as `None` in a field that accepts it, such as `Optional[str]`; to keep the `""` of an optional field, use parquet. In a `str` field whose default is `None`, given directly or by a factory, it also comes back as `None`. The empty string of a field whose factory reads the other fields is not restored: the empty cell counts as missing, and resuming fills it with the factory's value, even where the run saved `""`. A validator that turns an answer into `None` in a `str` field gives a value that the field's type rejects. In parquet, resuming flags that `None` for reprocessing. In CSV and XLSX, it flags the field when the validator turns the empty string itself into `None`, and returns `""` when it turns another answer, such as `"N/A"`, into `None`, because the empty cell does not say which of the two was saved. To keep that `None`, declare the field as `Optional[str]`. This holds for a run that applies the model's validators; in per-field mode (`search_per_field`), they do not run during execution, which saves the empty string as it came, and resuming from CSV and XLSX flags the field whose validator turns the empty string into `None`, while resuming from parquet returns the saved `""`. Parquet stores the empty string and `None` apart, and resuming returns the empty string as it was saved. In XLSX, text that starts with `=` is saved as text, not as a formula. XLSX stores numbers as doubles, and an integer above 2**53 comes back rounded; for ids that large, use CSV or parquet. None of the three formats keeps the NaN of a `float` field apart from null: CSV and XLSX write both as an empty cell, and parquet as null. When resuming, the NaN comes back missing, and a required field that received it is flagged for reprocessing. In parquet, an integer above 2**63 - 1 next to a pending row and an integer above 2**53 next to a float in the same column are not saved, and the checkpoint save fails with a warning. If execution is interrupted, run the same call again with the same input: with `resume=True`, the default, and the file present, execution continues from the checkpoint. On every save, `<checkpoint>.dataframeit.json` is written next to it with the run's signature (prompt, provider, model, `model_kwargs` except `timeout`, schema, search, and the text and status columns), a hash of each row's text, and a hash of the checkpoint itself. The checkpoint is resumed only when all three match; with another configuration, another input, or a file that is not the one the signature describes, the run warns, starts over, and overwrites it. In a row already concluded, successfully or with an error, the model fields come from the checkpoint, over whatever the input carried in them. To keep an output corrected by hand as it is, pass it with the status column, and the checkpoint is not read again. A complete output is returned without it, and the column comes back with `output["_dataframeit_status"] = "processed"` (or the name passed in `status_column`). With `resume=False`, the file is ignored and overwritten.

In the signature, the schema is the model's JSON Schema. When a field's `json_schema_extra`, nested fields included, holds a function, such as the condition from `field_condition`, or a set with a value that is not an integer, such as `{"in": {"pf", "mei"}}` in a `condition`, the attributes of each field take its place, with the set sorted and the function entered by its qualified name and its code: bytecode, constants and the names it reads. A `functools.partial` enters by the function it wraps and its fixed arguments; an object with `__call__`, by the code of `__call__` and the instance attributes, `__slots__` included; and a bound method, by the method's code and its owner: an object by its attributes, and the class of a `classmethod` by its name only. Editing the function, the `__call__`, the method or an argument of the `partial`, or changing an attribute of the object, changes the signature, and the run starts over. What the function reads from outside its own code does not enter: with another value in a global variable or in an enclosing scope, another default argument value or another body in a function it calls, the signature is the same, and resuming accepts the rows already concluded, in which the field was asked or skipped by the previous condition. To change the condition through one of these paths, pass `resume=False` or another `checkpoint_path`. The bytecode changes from one minor Python version to another, such as from 3.11 to 3.12, and the checkpoint of a model with a function is not resumed after that change.

Expand Down
2 changes: 1 addition & 1 deletion docs/en/reference/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -277,7 +277,7 @@ Identifies the fields of a Pydantic model that hold a structure instead of a sim
def get_complex_fields(pydantic_model) -> set
```

Returns the set of names of the fields whose type is a structure: `list`, `tuple`, `set`, `frozenset`, `dict`, `Sequence`, `AbstractSet` and `Mapping` and their mutable versions, `TypedDict`, `NamedTuple` or a nested Pydantic model, including inside `Optional` and `Union`.
Returns the set of names of the fields whose type is a structure: `list`, `tuple`, `deque`, `set`, `frozenset`, `dict`, `Sequence`, `AbstractSet` and `Mapping` and their mutable versions, `TypedDict`, `NamedTuple` or a nested Pydantic model, including inside `Optional` and `Union`.

```python
from dataframeit import get_complex_fields, normalize_complex_columns, read_df
Expand Down
Loading
Loading