Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feat](nereids) revert #23421: add between predicate back #48104

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

yujun777
Copy link
Collaborator

@yujun777 yujun777 commented Feb 19, 2025

What problem does this PR solve?

When parse SQL random(1, 10) between 2 and 8, it will rewrite as random(1, 10) >= 2 and random(1, 10) <= 8. But at the rewritten moment, the random is still an unbound function, and it can't be known that it's a non-foldable function. Then it will bound it to RANDOM function, but since it contains two unbound random functions, so it will got two distinct bounded RANDOM function, and the expression will evaluate random two times. It's wrong, so at the parse phase, BETWEEN shold not rewrite to AND. Only after bound functions, can rewrite BETWEEN to AND.

So we need to add between predicate back. Just revert #23421.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@yujun777
Copy link
Collaborator Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 31554 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9588b9a9b42daee915e2edc00d8a7a675e34e762, data reload: false

------ Round 1 ----------------------------------
q1	17589	5436	5145	5145
q2	2052	296	162	162
q3	10429	1245	749	749
q4	10215	997	547	547
q5	7551	2452	2259	2259
q6	200	170	134	134
q7	911	764	618	618
q8	9311	1314	1223	1223
q9	4795	4555	4684	4555
q10	6833	2314	1886	1886
q11	459	275	261	261
q12	351	370	230	230
q13	17769	3713	3095	3095
q14	232	227	204	204
q15	492	470	456	456
q16	620	619	584	584
q17	588	872	342	342
q18	6505	6096	6065	6065
q19	1207	949	561	561
q20	315	350	197	197
q21	2850	2139	1972	1972
q22	359	327	309	309
Total cold run time: 101633 ms
Total hot run time: 31554 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5213	5229	5379	5229
q2	239	330	234	234
q3	2139	2685	2342	2342
q4	1458	1858	1359	1359
q5	4236	4147	4171	4147
q6	216	166	126	126
q7	1862	1815	1656	1656
q8	2612	2593	2612	2593
q9	7229	7162	7088	7088
q10	3010	3202	2779	2779
q11	577	530	496	496
q12	700	768	603	603
q13	3604	3853	3346	3346
q14	283	305	281	281
q15	509	484	463	463
q16	630	696	645	645
q17	1165	1546	1416	1416
q18	7500	7442	7362	7362
q19	853	965	1113	965
q20	1984	2025	1883	1883
q21	5485	5039	4687	4687
q22	613	573	524	524
Total cold run time: 52117 ms
Total hot run time: 50224 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 189840 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9588b9a9b42daee915e2edc00d8a7a675e34e762, data reload: false

query1	1295	959	947	947
query2	6216	1835	1821	1821
query3	10987	4392	4237	4237
query4	56010	24253	23349	23349
query5	5143	585	519	519
query6	369	210	187	187
query7	5130	516	298	298
query8	320	250	249	249
query9	6023	2557	2578	2557
query10	418	297	247	247
query11	15243	15082	14897	14897
query12	158	105	103	103
query13	1120	536	374	374
query14	10274	6674	6358	6358
query15	187	202	171	171
query16	7010	684	468	468
query17	1062	722	538	538
query18	1501	392	301	301
query19	195	193	174	174
query20	130	128	161	128
query21	211	124	115	115
query22	4489	4439	4275	4275
query23	33910	33125	33319	33125
query24	5661	2496	2454	2454
query25	480	468	402	402
query26	637	287	166	166
query27	1663	486	346	346
query28	3010	2480	2468	2468
query29	606	622	449	449
query30	210	196	157	157
query31	943	870	794	794
query32	75	72	68	68
query33	459	392	344	344
query34	783	881	538	538
query35	824	839	768	768
query36	968	970	877	877
query37	136	102	77	77
query38	4319	4307	4321	4307
query39	1495	1451	1458	1451
query40	208	122	111	111
query41	64	57	54	54
query42	126	107	113	107
query43	518	514	478	478
query44	1362	834	815	815
query45	181	175	171	171
query46	936	1112	684	684
query47	1818	1888	1800	1800
query48	399	450	319	319
query49	701	503	402	402
query50	757	771	428	428
query51	4247	4335	4226	4226
query52	109	109	102	102
query53	232	271	201	201
query54	493	508	424	424
query55	87	82	81	81
query56	268	315	246	246
query57	1154	1172	1134	1134
query58	256	239	236	236
query59	2822	2795	2597	2597
query60	300	277	282	277
query61	123	115	116	115
query62	762	733	688	688
query63	242	198	201	198
query64	1403	1056	693	693
query65	3395	3252	3138	3138
query66	762	400	333	333
query67	15824	15504	15375	15375
query68	6090	813	517	517
query69	550	298	269	269
query70	1208	1137	1131	1131
query71	455	291	269	269
query72	5356	3589	3726	3589
query73	1090	795	363	363
query74	9180	9169	9051	9051
query75	3217	3136	2687	2687
query76	3971	1173	744	744
query77	548	398	280	280
query78	10210	10351	9357	9357
query79	1880	892	615	615
query80	846	541	441	441
query81	531	281	241	241
query82	401	130	96	96
query83	194	174	152	152
query84	298	97	72	72
query85	754	361	297	297
query86	421	299	282	282
query87	4448	4393	4357	4357
query88	3012	2235	2204	2204
query89	407	319	285	285
query90	1711	194	195	194
query91	138	137	109	109
query92	76	64	62	62
query93	2465	1032	578	578
query94	713	421	292	292
query95	346	270	260	260
query96	499	559	275	275
query97	2804	2830	2742	2742
query98	243	207	205	205
query99	1303	1394	1287	1287
Total cold run time: 294584 ms
Total hot run time: 189840 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.14 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9588b9a9b42daee915e2edc00d8a7a675e34e762, data reload: false

query1	0.03	0.03	0.03
query2	0.09	0.03	0.04
query3	0.24	0.06	0.07
query4	1.61	0.10	0.10
query5	0.40	0.40	0.40
query6	1.16	0.66	0.67
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.60	0.54	0.53
query10	0.57	0.60	0.57
query11	0.15	0.11	0.11
query12	0.13	0.11	0.11
query13	0.61	0.59	0.61
query14	2.67	2.71	2.70
query15	0.91	0.86	0.86
query16	0.36	0.38	0.37
query17	1.01	1.01	1.03
query18	0.21	0.20	0.19
query19	1.90	1.77	1.98
query20	0.01	0.02	0.02
query21	15.36	0.90	0.53
query22	0.76	1.30	0.80
query23	14.74	1.38	0.63
query24	11.92	1.17	0.42
query25	0.34	0.09	0.18
query26	0.97	0.18	0.14
query27	0.06	0.05	0.05
query28	6.13	0.75	0.43
query29	12.51	3.93	3.27
query30	0.24	0.08	0.06
query31	2.83	0.56	0.38
query32	3.22	0.54	0.46
query33	3.14	2.99	2.94
query34	15.63	5.17	4.49
query35	4.53	4.51	4.54
query36	0.65	0.50	0.49
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.03
query40	0.18	0.14	0.12
query41	0.09	0.03	0.02
query42	0.03	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.26 s
Total hot run time: 30.14 s

@yujun777
Copy link
Collaborator Author

run buildall

@yujun777 yujun777 force-pushed the add-back-between-predicate branch from 097e500 to e0e2ddd Compare February 24, 2025 01:24
@yujun777
Copy link
Collaborator Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 31365 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e0e2ddd5fd5051ead83464c6d24de5ac99c27153, data reload: false

------ Round 1 ----------------------------------
q1	16662	5334	5168	5168
q2	2050	304	173	173
q3	10486	1342	720	720
q4	10418	986	519	519
q5	7491	2439	2273	2273
q6	202	175	139	139
q7	917	749	634	634
q8	9306	1313	1150	1150
q9	4835	4629	4589	4589
q10	6801	2330	1880	1880
q11	466	280	253	253
q12	353	364	218	218
q13	17787	3649	3058	3058
q14	239	231	211	211
q15	504	448	456	448
q16	614	618	577	577
q17	585	868	342	342
q18	6439	6100	6204	6100
q19	874	946	548	548
q20	317	330	193	193
q21	2715	2106	1862	1862
q22	355	327	310	310
Total cold run time: 100416 ms
Total hot run time: 31365 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5127	5165	5145	5145
q2	241	331	240	240
q3	2165	2678	2305	2305
q4	1391	1814	1363	1363
q5	4228	4111	4122	4111
q6	228	162	125	125
q7	1876	1813	1666	1666
q8	2504	2496	2439	2439
q9	6810	6721	6704	6704
q10	2879	3062	2647	2647
q11	575	498	474	474
q12	655	755	566	566
q13	3254	3667	3093	3093
q14	274	285	275	275
q15	503	470	470	470
q16	629	645	607	607
q17	1100	1588	1290	1290
q18	7171	6932	6875	6875
q19	757	781	921	781
q20	1886	1944	1815	1815
q21	5096	4776	4701	4701
q22	612	557	535	535
Total cold run time: 49961 ms
Total hot run time: 48227 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 183203 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e0e2ddd5fd5051ead83464c6d24de5ac99c27153, data reload: false

query1	978	402	362	362
query2	6527	1898	1906	1898
query3	6786	210	210	210
query4	25912	23613	22899	22899
query5	5085	674	487	487
query6	297	199	175	175
query7	4600	496	293	293
query8	286	244	229	229
query9	8617	2572	2576	2572
query10	475	309	265	265
query11	15795	14981	14794	14794
query12	172	106	107	106
query13	1668	516	401	401
query14	11542	6313	6440	6313
query15	238	198	173	173
query16	7723	604	486	486
query17	1512	715	535	535
query18	1982	390	298	298
query19	256	200	155	155
query20	123	117	120	117
query21	210	124	100	100
query22	4536	4314	4497	4314
query23	33687	33009	32914	32914
query24	6913	2353	2376	2353
query25	477	449	378	378
query26	1205	274	153	153
query27	2154	488	339	339
query28	3977	2413	2390	2390
query29	694	540	432	432
query30	221	184	162	162
query31	938	817	740	740
query32	73	62	77	62
query33	557	352	297	297
query34	748	831	503	503
query35	790	815	745	745
query36	928	967	890	890
query37	123	98	72	72
query38	4249	4181	4092	4092
query39	1451	1417	1398	1398
query40	202	118	104	104
query41	54	50	51	50
query42	122	104	103	103
query43	502	513	474	474
query44	1277	777	780	777
query45	176	175	160	160
query46	851	1039	645	645
query47	1780	1804	1708	1708
query48	402	413	300	300
query49	807	516	427	427
query50	691	743	425	425
query51	4163	4199	4122	4122
query52	108	107	97	97
query53	227	253	190	190
query54	502	527	410	410
query55	119	78	79	78
query56	253	260	257	257
query57	1116	1160	1063	1063
query58	243	240	251	240
query59	2647	2673	2598	2598
query60	288	271	255	255
query61	148	123	158	123
query62	793	739	672	672
query63	233	187	190	187
query64	4205	980	652	652
query65	3239	3164	3161	3161
query66	1066	396	300	300
query67	15807	15615	15376	15376
query68	7818	755	509	509
query69	460	301	266	266
query70	1157	1129	1045	1045
query71	420	298	268	268
query72	5738	3583	3735	3583
query73	697	749	358	358
query74	9210	9192	9005	9005
query75	3174	3168	2683	2683
query76	3277	1173	732	732
query77	543	369	289	289
query78	9830	9869	9362	9362
query79	3125	824	586	586
query80	602	533	438	438
query81	492	281	234	234
query82	708	122	96	96
query83	174	170	164	164
query84	243	93	69	69
query85	801	362	296	296
query86	377	307	305	305
query87	4501	4543	4446	4446
query88	4107	2220	2210	2210
query89	413	323	296	296
query90	1976	193	196	193
query91	140	142	113	113
query92	77	60	58	58
query93	2044	1001	566	566
query94	684	423	268	268
query95	348	273	260	260
query96	493	533	276	276
query97	2831	2842	2720	2720
query98	234	215	194	194
query99	1321	1400	1259	1259
Total cold run time: 274541 ms
Total hot run time: 183203 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.21 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e0e2ddd5fd5051ead83464c6d24de5ac99c27153, data reload: false

query1	0.03	0.04	0.03
query2	0.07	0.03	0.04
query3	0.23	0.07	0.07
query4	1.61	0.10	0.10
query5	0.42	0.42	0.41
query6	1.17	0.67	0.65
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.59	0.52	0.51
query10	0.58	0.57	0.57
query11	0.15	0.11	0.11
query12	0.15	0.11	0.11
query13	0.63	0.60	0.60
query14	2.72	2.71	2.71
query15	0.92	0.86	0.85
query16	0.36	0.37	0.38
query17	1.01	1.00	1.02
query18	0.22	0.19	0.20
query19	1.90	1.79	1.96
query20	0.01	0.01	0.01
query21	15.35	0.88	0.54
query22	0.75	1.15	0.61
query23	15.04	1.36	0.60
query24	7.25	1.55	0.56
query25	0.47	0.24	0.13
query26	0.71	0.15	0.14
query27	0.06	0.05	0.05
query28	9.16	0.86	0.41
query29	12.54	3.94	3.30
query30	0.25	0.08	0.05
query31	2.86	0.58	0.37
query32	3.22	0.55	0.48
query33	3.02	3.03	3.03
query34	15.73	5.08	4.48
query35	4.50	4.50	4.56
query36	0.67	0.52	0.49
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.03	0.02
query40	0.17	0.14	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 104.9 s
Total hot run time: 30.21 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants