mirror/ ultrajson
1
0
Fork 0
mirror of https://github.com/ultrajson/ultrajson.git synced 2024-05-07 13:56:13 +02:00
Code Issues
916 Commits 3 Branches 26 Tags 8.6 MiB
Commit Graph

916 Commits

Author SHA1 Message Date
JustAnotherArchivist 98321fad98 Switch to NULL encoding (= UTF-8) to avoid string comparison in PyUnicode_AsEncodedString 2022-05-30 01:58:58 +00:00
JustAnotherArchivist 9b9af1ab70 Fix handling of surrogates on encoding 
This allows surrogates anywhere in the input, compatible with the json module from the standard library.

This also refactors two interfaces:
- The `PyUnicode` to `char*` conversion is moved into its own function, separated from the `JSONTypeContext` handling, so it can be reused for other things in the future (e.g. indentation and separators) which don't have a type context.
- Converting the `char*` output to a Python string with surrogates intact requires the string length for `PyUnicode_Decode` & Co. While `strlen` could be used, the length is already known inside the encoder, so the encoder function now also takes an extra `size_t` pointer argument to return that and no longer NUL-terminates the string. This also permits output that contains NUL bytes (even though that would be invalid JSON), e.g. if an object's `__json__` method return value were to contain them.

Fixes #156
Fixes #447
Fixes #537
Supersedes #284
 2022-05-30 01:58:12 +00:00
Philipp Otterbein 0a0e111701 fix typo: LLONG_MIN 2022-05-29 19:24:07 +02:00
Philipp Otterbein 9c42263c80 fix integer decoding: always detect overflows 2022-05-28 17:01:30 +02:00
Philipp Otterbein cb180ce34c add additional tests for integer overflow during parsing 2022-05-28 16:10:15 +02:00
Hugo van Kemenade b300d642f6
Merge pull request #540 from hugovk/fix-travis-links 2022-05-18 22:33:58 +03:00
Hugo van Kemenade 9e8203253f Fix Travis CI links 2022-05-18 22:03:44 +03:00
Hugo van Kemenade 80d0bd41a2
Merge pull request #539 from hugovk/add-3.11 2022-05-18 21:58:24 +03:00
Hugo van Kemenade 4b0a625dcc Test Python 3.11 beta 2022-05-18 15:15:12 +03:00
Hugo van Kemenade 6bba49f50b
Merge pull request #535 from ultrajson/all-repos_autofix_all-repos-sed 2022-05-13 11:29:57 +03:00
Hugo van Kemenade 1a7a5f19ce Remove redundant wheel dependency from pyproject.toml 
Committed via https://github.com/asottile/all-repos
 2022-05-04 11:17:40 +03:00
Hugo van Kemenade ebdb150991
Merge pull request #534 from Erotemic/fix-ci-benchmark 
Benchmark CI fixes
 2022-04-27 13:06:04 +03:00
joncrall 4e527b84b0
remove paren 2022-04-26 17:14:06 -04:00
joncrall 0e280117f2
Spelling 2022-04-26 12:32:37 -04:00
joncrall f63a3e471e
Spelling 2022-04-26 12:31:49 -04:00
Jon Crall c5d2a07ee6
Update tests/benchmark.py 
Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
 2022-04-26 12:30:37 -04:00
joncrall 2c9e70f965
Run benchmarks in CI when they are changed 2022-04-25 14:11:11 -04:00
joncrall e8fc9e1190
Add proper default to disable 2022-04-25 14:08:03 -04:00
Hugo van Kemenade a900e469f9
Merge pull request #533 from Erotemic/benchmark-flexibility 2022-04-25 12:09:01 +03:00
JustAnotherArchivist 66060a0fab Add and fix debug memory checks for all buffer appends on encoding 
In DEBUG mode, this ensures that all buffer appends are safe.
It also refactors direct `memcpy` calls into a helper `Buffer_memcpy` function that ensures correct buffer pointer movement and has a similar safety check.
 2022-04-24 13:58:00 -07:00
joncrall 3d25fb4fe6
remove skip-lib-comps command 2022-04-22 18:57:46 -04:00
joncrall 991bf12059
Merge branch 'benchmark-flexibility' of github.com:Erotemic/ultrajson into benchmark-flexibility 2022-04-21 17:34:06 -04:00
joncrall 3850b93182
Remove cpuinfo 2022-04-21 17:33:51 -04:00
pre-commit-ci[bot] 3eb7a19f60 [pre-commit.ci] auto fixes from pre-commit.com hooks 
for more information, see https://pre-commit.ci
 2022-04-21 21:13:27 +00:00
joncrall eb71ee87c5
Merge branch 'benchmark-flexibility' of github.com:Erotemic/ultrajson into benchmark-flexibility 2022-04-21 16:59:41 -04:00
pre-commit-ci[bot] 79a6122e5f [pre-commit.ci] auto fixes from pre-commit.com hooks 
for more information, see https://pre-commit.ci
 2022-04-21 20:59:32 +00:00
joncrall 78f24e290e
Add note about units of benchmarks 2022-04-21 16:57:05 -04:00
joncrall 32de298ee7
Added argparse CLI with ability to disable specific modules 2022-04-21 16:39:34 -04:00
joncrall ccd30400b0
Generalize the way new json modules can be added to existing benchmarks 2022-04-21 16:16:22 -04:00
JustAnotherArchivist b3f8754c8a Fix segmentation faults when handling unserialisable objects 
Errors during `__repr__` itself as well as ones during the conversion to a bytes object were not handled, resulting in NULL pointer dereferencing.

Cf. #382
 2022-04-18 12:20:18 +01:00
Hugo van Kemenade 779949896e
Merge pull request #526 from JustAnotherArchivist/fix-obj-str-exception-segv 
Fix segmentation fault when an exception is raised while converting a dict key to a string
 2022-04-14 07:00:55 +03:00
Hugo van Kemenade b8a85c8488
Merge pull request #527 from JustAnotherArchivist/fuzz-ref-count 
Add ref count checks to fuzz tests
 2022-04-13 21:55:22 +03:00
Hugo van Kemenade 19c21d4801
Merge pull request #528 from JustAnotherArchivist/test-sorted-key-ref-counts 
Add test for key ref counts with sort_keys enabled
 2022-04-13 08:59:34 +03:00
JustAnotherArchivist 97fceb6245 Add test for key ref counts with sort_keys enabled 2022-04-13 03:02:55 +00:00
JustAnotherArchivist 9ac2133e0f Add ref count checks to fuzz tests 2022-04-13 02:52:17 +00:00
JustAnotherArchivist 935fe0cec4 Fix segmentation fault when an exception is raised while converting a dict key to a string 
Fixes #522
 2022-04-13 00:04:24 +00:00
JustAnotherArchivist 62dec8de71 Fix ref counting on non-string dict keys 
For bytes, there was an extraneous INCREF; PyIter_Next returns a new reference. For other non-strings, the original itemName before converting to a string was never dereferenced.

Fixes #419
 2022-04-07 20:31:36 +01:00
JustAnotherArchivist 2d1f088c2e Fix ref counting on repeated default function calls 
Fixes #523
 2022-04-07 20:20:01 +01:00
JustAnotherArchivist f6860f1f3d Remove shebang 
Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
 2022-04-05 21:04:39 +01:00
JustAnotherArchivist c0ff7b1df9 python -m pytest 
Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
 2022-04-05 21:04:39 +01:00
JustAnotherArchivist 362fed3cdc Clearer pytest command 
Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
 2022-04-05 21:04:39 +01:00
JustAnotherArchivist 82917c0cb1 actions/checkout@v3 
Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
 2022-04-05 21:04:39 +01:00
JustAnotherArchivist 3c095f1912 Widen tests to cover more possible buffer overflows 
If the default output format changes in the future (e.g. `separators` as in the standard library), these tests would otherwise become irrelevant.
 2022-04-05 21:04:39 +01:00
JustAnotherArchivist f4d2c87ab6 Refactor buffer reservations to ensure sufficient space on all additions 
* Removed the reservations in Buffer_EscapeStringUnvalidated and Buffer_EscapeStringValidated as those are not needed and may hide other bugs.
* Debug check in Buffer_EscapeStringValidated was triggering incorrectly.
* The reservation on JT_RAW was much larger than necessary; the value is copied directly, so the factor six is not needed, and this may hide other bugs.
* Explicit accurate reservations everywhere else.
 2022-04-05 21:04:39 +01:00
Brénainn Woodsend 1846e082a1 Add fuzz test to CI/CD. 2022-04-05 21:04:39 +01:00
Brénainn Woodsend 5875168c41 Fix some more seg-faults on encoding. 2022-04-05 21:04:39 +01:00
Brénainn Woodsend 1a39406b3a Remove the hidden JSON_NO_EXTRA_WHITESPACE compile knob. 
Unsetting it can lead to seg-faults. I don't think it's worth having to fix and
then test this undocumented permutation.
 2022-04-05 21:04:39 +01:00
Brénainn Woodsend 20aa1a69e5 Add a fuzzing test to search for segfaults in encoding. 2022-04-05 21:04:39 +01:00
Brénainn Woodsend 61dd6f19e8 Fix unchecked buffer overflows (CVE-2021-45958). 
Add a few extra memory reserve calls to account for the extra space that
indentation needs.

These kinds of memory issues are hard to spot because the buffer is resized in
powers of 2 meaning that a miscalculation would only show any symptoms if the
required buffer size is estimated to be just below a 2 power but is actually
just above. Add a debug mode which replaces the 2 power scheme with reserving
only the memory explicitly requested and adds some overflow checks.
 2022-04-05 21:04:39 +01:00
Hugo van Kemenade 881ee9317b
Merge pull request #514 from Erotemic/nan_loads 2022-04-04 23:29:22 +03:00