[Bug]: [benchmark][standalone] TimeTick Lag is very high, causing DQL request timeout #36195
Comments
wangting0128
added
kind/bug
yanliang567
added
triage/accepted
the cpu is full and request takes hours. |
|
as long as service didn't crash i thought it's fine. |
The normal average DQL time is < 500ms. During the DQL request timeout(60s) period, the CPU is not fully utilized. I think this should be a problem that needs to be checked. 🤔️
|
|
Only 2M of data was inserted, but the Queryable Entity Num showed 44.8M, and the memory increased from 5G to 57+G. 
|
How did you define only 2M is inserted? is seems that you have delete and insert in the test. I think most of the 44.8M data has been deleted but not compacted in time. is this what you are trying to test? the compaction can catch up with deletes |
|
|
Here's what I see: search timeout(but cannot cancel)-> pining segments -> memory and segment count raising
|
|
This is how search works: if thery are submiited into c++, when golang timeout and returned for like 1min, the c++ part will continuous to run.
CPU is down when all search/query in c++ finished. In the mean time ,some of the search/query finished during this short time. |
|
The behavior is expected, nothing abnormal, except perhaps we need smaller search tasks that took less than 1hrs. Also, we'might need to be able to cancel c++ tasks from golang side to aviod such long-time pin. /unassign |
|
@wangting0128 In your tests, from the metrics, it's more likely there're 99% of VERY LONG DQL with 1% of quick DQL. |
|
|
doesn't seen reasonable, I'll look into this. |
|
|
|
verified scalar fields compare scalar fields not build index scalar build INVERTED index scalar build BITMAP index |
|
maybe we should change bitmap index to hasRawData = false? |
|
also for compare, if one column has index, we could change to iterate with one row and check the other row. But anyway compare should not be a very important op we spend time with |
if changed to false, will load raw data again , and this will increase memory cost and a heavy operation. I think compare and reverse_up is not a high use frequency operation, if latency too much, can use cache param |
|
We need add a new attribute on index, retreive_enabled. |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
Is there an existing issue for this?
Environment
Current Behavior
argo task：fouramf-bitmap-scenes-fdgrx
test case name：test_bitmap_locust_dql_dml_standalone
server：
client test result:
Expected Behavior
No response
Steps To Reproduce
concurrent test and calculation of RT and QPS :purpose: `primary key: INT64 autoID` 1. building `BITMAP` index on all supported 12 scalar fields 2. 2 fields of different vector types 3. verify DQL & DML requests :test steps: 1. create collection with fields: 'float_vector': 128dim 'sparse_float_vector': sparse_range=[1, 100] <- the range of non-zero values of a sparse vector 'id': primary key type is INT64 all scalar fields: varchar max_length=100, array max_capacity=13 2. build indexes: IVF_SQ8: 'float_vector' SPARSE_WAND: 'sparse_float_vector' BITMAP: all scalar fields 3. insert 2 million data 4. flush collection 5. build indexes again using the same params 6. load collection 7. concurrent request: - search - query - hybrid_search - load - insert - delete: delete all inserted data - flush: ignore RateLimiterMilvus Log
No response
Anything else?
No response
The text was updated successfully, but these errors were encountered: