Modernize deps & optimize matching engine performance#401
Open
lispc wants to merge 18 commits into
Open
Conversation
Also make persist internals pub(crate) for unit tests
This reverts commit dc3b28b.
…submodule dependency
…y, market data/K-line, state recovery)
…ly signature, market order adaptive check, longer persistor wait)
- Fix tests/trade.ts import path ./fluidex -> ../fluidex - Fix all new e2e tests initAccounts to register users sequentially (server overrides user_id to last_user_id+1) - Fix balance_consistency.ts precision test: server rounds, not rejects - Fix state_recovery.ts debugReset test: verify clean state after hard reset - Includes docker-compose.yaml inline fix (remove orchestra submodule dep)
This makes the matching engine usable as a standalone library without any zk-specific dependencies. Key changes: - New module with trait abstractions: - : pluggable order signature verification - : pluggable order commitment builder - implementations for non-zk deployments - implementations (BabyJubJub + Poseidon) for zk-rollup - : feature gates babyjubjub-rs, poseidon-rs, ff - : module hidden behind feature - : removed hard-coded - : uses injected trait objects - : - now accepts + - changed to - : removed (moved to ) - : removed (moved to ) - : selects auth impl based on feature Non-zk build: Zk build: (default features include zk-rollup) All 40 unit tests pass in both configurations. Clippy clean in both configurations.
Non-zk deployments are now the default. To build with zk support:
cargo build --features zk-rollup
All Rust unit tests (40/40) and JS e2e tests pass in both modes.
Prevents syntax error 'limit' that caused /internal_txs endpoint to return empty results, breaking transfer.ts integration tests.
- Fix CI workflow names (all were 'CI') and update branch protection rules to remove stale '1.56.0' suffix from required status checks - Update docs (testing.md, AGENTS.md, README.md): - Correct perftest CLI params (--requests, --duration-secs) - Document pair-trade sustainability (periodic cancelAll + role swap) - Add Node.js vs Rust perf comparison (~100x gap) - Add macOS advisory lock workaround note - Rewrite stress.ts: dual-user, periodic cancel, total balance check - Add perf_test.ts: 50-user mixed-load stress test - perftest.rs: add periodic cancelAll + role swap for sustainable pair-trade - persistor.rs: use dedicated PgConnection for migration (macOS advisory lock fix) - dto.rs: downgrade empty signature log from warn to debug
…lanceMapKey hashing - Kafka producer channel capacity 100K -> 1M (avoid panic under load) - Trade market/base/quote: String -> InternedString - Replace serde_json with simd-json for Kafka message serialization - BalanceMapKey.asset: String -> InternedString with pointer-based Hash - InternedString: add From<&str>, Eq, Hash impls
lispc
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR modernizes multiple parts of the codebase and adds a round of matching-engine performance optimizations.
Modernization Changes
tonic-buildinbuild.rsinstead of hand-copiedsrc/rpc/exchange/andsrc/rpc/rollup/BaseProducermanual thread withFutureProducer+tokio::sync::mpscchannelmessage/trait hierarchy: replacedRdConsumerExt,MessageHandlerAsync,TypedMessageHandlerAsync,TypedMessageHandler,SyncTyped,Simple,TopicBuilder,ChainedTopicBuilderwith a cleanMessageHandlertrait +TypedHandler/SimpleHandlerfluidex-commongit dependency entirely; inline crypto traits (FrExt,DecimalExt,PubkeyExt,SignatureExt) intosrc/utils/crypto.rsbabyjubjub-rs,ff_ce,poseidon-rs,num-bigint,num-traitsorchestra/docker→docker/protobuf-compilerfortonic-buildPerformance Optimizations
Profiling under pair-trade load revealed the engine was bottlenecked by Kafka producer backpressure and JSON serialization.
try_send().unwrap()panic under loadTradestruct:market/base/quotefromStringtoInternedString, eliminating per-trade heap allocationsserde_json::to_stringwithsimd_json::to_stringBalanceMapKey:assetfromStringtoInternedString; hash on the interned pointer for O(1)HashMaplookupsInternedString: addFrom<&str>,PartialEq,Eq, andHashimpls needed by the new callersKnown Limitations / Notes
mpscconsumer). The measured single-market throughput ceiling is ~10K orders/s with the current client setup.simd-jsonis used only for Kafka producer serialization, not for REST/gRPC payloads, so downstream JSON consumers remain compatible.