[opt](metadata) optimize lock in recycle bin

[morrySnow]

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

This pull request refactors the CatalogRecycleBin class to improve thread safety and concurrency. The main changes involve replacing synchronized methods and non-thread-safe maps with concurrent data structures and explicit locking, ensuring safer access and modification in multi-threaded scenarios.

Thread safety improvements:

• Replaced HashMap fields (idToDatabase, idToTable, idToPartition, idToRecycleTime) with thread-safe ConcurrentMap variants to allow concurrent access.
• Introduced a ReadWriteLock (lock) to manage concurrent read/write access, and wrapped critical sections of code that modify shared state with appropriate lock acquisition and release. [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21]

Code modernization and cleanup:

• Removed the use of the synchronized keyword from method signatures, relying instead on the new explicit locking mechanism for thread safety. [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21]
• Updated imports to include necessary concurrency and testing annotations. [1] [2]
• Annotated allTabletsInRecycledStatus with @VisibleForTesting and removed its synchronization for better testability and clarity.

These changes make the recycle bin operations safer and more robust in concurrent environments, which is crucial for a multi-threaded system like Doris.

Release note

None

Check List (For Author)

• Test

  • Regression test
  • Unit Test
  • Manual test (add detailed scripts or steps below)
  • No need to test or manual test. Explain why:
    • This is a refactor/code format and no logic has been changed.
    • Previous test can cover this change.
    • No code files have been changed.
    • Other reason

• Behavior changed:

  • No.
  • Yes.

• Does this need documentation?

  • No.
  • Yes.

Check List (For Reviewer who merge this PR)

• Confirm the release note
• Confirm test cases
• Confirm document
• Add branch pick label

[Thearas]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[morrySnow]

run buildall

[doris-robot]

TPC-H: Total hot run time: 32036 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d279a31c536f9e23a8bbbc908f911492e6dc4b90, data reload: false

------ Round 1 ----------------------------------
q1	17633	4273	4052	4052
q2	2024	353	246	246
q3	10172	1261	712	712
q4	10197	817	304	304
q5	7499	2081	1838	1838
q6	184	175	138	138
q7	963	775	653	653
q8	9272	1408	1109	1109
q9	5123	4669	4549	4549
q10	6822	1799	1410	1410
q11	502	300	287	287
q12	754	749	586	586
q13	17772	3869	3159	3159
q14	287	292	281	281
q15	595	516	501	501
q16	678	696	633	633
q17	660	813	524	524
q18	7215	6559	6857	6559
q19	1432	1009	654	654
q20	431	395	270	270
q21	3295	2608	2518	2518
q22	1123	1070	1053	1053
Total cold run time: 104633 ms
Total hot run time: 32036 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4505	4259	4252	4252
q2	339	411	342	342
q3	2329	2709	2404	2404
q4	1462	1948	1516	1516
q5	4631	4409	4439	4409
q6	211	168	141	141
q7	1975	1897	1770	1770
q8	2583	2425	2573	2425
q9	7255	7093	7041	7041
q10	2512	2614	2248	2248
q11	553	472	454	454
q12	708	804	821	804
q13	3559	3882	3105	3105
q14	282	287	258	258
q15	522	500	485	485
q16	626	659	621	621
q17	1124	1332	1295	1295
q18	7383	7220	7190	7190
q19	811	803	812	803
q20	1878	1965	1845	1845
q21	4519	4186	4234	4186
q22	1038	1031	986	986
Total cold run time: 50805 ms
Total hot run time: 48580 ms

[hello-stephen]

FE UT Coverage Report

Increment line coverage 67.83% (234/345) 🎉
Increment coverage report
Complete coverage report

[doris-robot]

TPC-DS: Total hot run time: 173238 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d279a31c536f9e23a8bbbc908f911492e6dc4b90, data reload: false

query5	4838	612	464	464
query6	337	230	217	217
query7	4221	452	250	250
query8	328	248	232	232
query9	8737	2893	2840	2840
query10	519	374	316	316
query11	15334	15238	14909	14909
query12	167	112	115	112
query13	1239	470	364	364
query14	6107	2996	2796	2796
query14_1	2678	2693	2665	2665
query15	200	188	173	173
query16	990	480	473	473
query17	1038	697	565	565
query18	2429	423	363	363
query19	219	216	186	186
query20	122	113	114	113
query21	214	135	120	120
query22	3957	4101	3811	3811
query23	15960	15644	15387	15387
query23_1	15397	15357	15485	15357
query24	7201	1539	1154	1154
query24_1	1163	1146	1138	1138
query25	528	436	396	396
query26	1012	260	171	171
query27	2750	441	267	267
query28	4568	2120	2123	2120
query29	781	510	423	423
query30	300	240	214	214
query31	774	617	567	567
query32	81	74	75	74
query33	529	345	320	320
query34	890	865	514	514
query35	745	775	684	684
query36	892	882	824	824
query37	135	100	89	89
query38	2710	2635	2581	2581
query39	764	767	739	739
query39_1	708	728	731	728
query40	231	142	126	126
query41	74	71	67	67
query42	103	102	104	102
query43	475	507	423	423
query44	1336	761	740	740
query45	188	189	186	186
query46	863	958	564	564
query47	1433	1539	1311	1311
query48	311	325	239	239
query49	596	474	348	348
query50	616	277	197	197
query51	3777	3788	3788	3788
query52	103	106	96	96
query53	284	316	268	268
query54	291	269	279	269
query55	81	81	77	77
query56	313	305	321	305
query57	1040	991	940	940
query58	272	270	267	267
query59	2143	2214	2075	2075
query60	340	330	322	322
query61	159	150	159	150
query62	412	355	317	317
query63	302	262	277	262
query64	4695	1291	980	980
query65	3835	3729	3725	3725
query66	1418	414	323	323
query67	15268	15355	14641	14641
query68	2637	1028	727	727
query69	464	368	327	327
query70	1020	905	931	905
query71	317	308	287	287
query72	5576	3342	3569	3342
query73	619	710	313	313
query74	8824	8693	8557	8557
query75	2760	2849	2495	2495
query76	2401	1087	650	650
query77	366	389	311	311
query78	9802	9850	9233	9233
query79	1055	908	568	568
query80	1035	616	521	521
query81	531	269	240	240
query82	1369	146	115	115
query83	372	259	255	255
query84	258	120	102	102
query85	1049	620	448	448
query86	369	309	307	307
query87	2899	2896	2696	2696
query88	3478	2583	2555	2555
query89	392	348	329	329
query90	1794	170	165	165
query91	175	163	144	144
query92	75	77	70	70
query93	912	905	516	516
query94	589	295	298	295
query95	569	389	319	319
query96	642	498	227	227
query97	2346	2386	2286	2286
query98	228	201	192	192
query99	616	597	520	520
Total cold run time: 247647 ms
Total hot run time: 173238 ms

[doris-robot]

ClickBench: Total hot run time: 26.75 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d279a31c536f9e23a8bbbc908f911492e6dc4b90, data reload: false

query1	0.06	0.04	0.04
query2	0.09	0.04	0.04
query3	0.26	0.08	0.08
query4	1.61	0.12	0.11
query5	0.27	0.25	0.26
query6	1.15	0.68	0.65
query7	0.03	0.03	0.03
query8	0.06	0.04	0.04
query9	0.57	0.50	0.50
query10	0.54	0.55	0.56
query11	0.15	0.09	0.09
query12	0.14	0.10	0.11
query13	0.59	0.57	0.59
query14	0.95	0.96	0.95
query15	0.79	0.77	0.79
query16	0.40	0.39	0.39
query17	1.07	1.05	1.04
query18	0.23	0.21	0.21
query19	1.86	1.90	1.75
query20	0.02	0.01	0.01
query21	15.46	0.27	0.14
query22	5.28	0.05	0.04
query23	16.02	0.27	0.11
query24	0.95	0.60	0.32
query25	0.11	0.11	0.05
query26	0.13	0.14	0.14
query27	0.08	0.06	0.05
query28	4.50	1.05	0.89
query29	12.53	3.88	3.15
query30	0.28	0.16	0.14
query31	2.82	0.60	0.39
query32	3.24	0.56	0.47
query33	2.96	3.01	2.98
query34	16.18	5.06	4.45
query35	4.42	4.52	4.40
query36	0.65	0.50	0.50
query37	0.11	0.06	0.07
query38	0.07	0.04	0.04
query39	0.04	0.03	0.03
query40	0.17	0.15	0.13
query41	0.09	0.03	0.04
query42	0.05	0.03	0.03
query43	0.05	0.04	0.04
Total cold run time: 97.03 s
Total hot run time: 26.75 s