Hello, thank you for your excellent open-source work. I am trying to reproduce your code using python main.py --train --base configs/stableSRNew/v2-finetune_text_T_512.yaml --gpus GPU_ID, --name NAME --scale_lr False.
Here's my training log. It's running on 6 A30 GPUs with max steps set to 100,000. Based on calculations, training will likely take about 10 days, which far exceeds the training time for stable_000117 you mentioned in #36. What issues might I have overlooked?
train/loss_simple_step,train/loss_vlb_step,train/loss_step,global_step,epoch,created_at
0.25144094228744507,0.001727868104353547,0.25144094228744507,49.0,0,2025-11-07 06:02:10.484610
0.0960497185587883,0.0004331176169216633,0.0960497185587883,99.0,0,2025-11-07 06:09:01.771131
0.10094963759183884,0.00045272731222212315,0.10094963759183884,149.0,0,2025-11-07 06:15:52.570149
0.079338937997818,0.0004949959111399949,0.079338937997818,199.0,0,2025-11-07 06:22:42.938209
0.09198877215385437,0.0003369307960383594,0.09198877215385437,249.0,0,2025-11-07 06:29:33.508085
0.16970014572143555,0.002384641207754612,0.16970014572143555,299.0,0,2025-11-07 06:36:23.734713
0.07759910821914673,0.0002961623831652105,0.07759910821914673,349.0,0,2025-11-07 06:43:13.754978
0.07155339419841766,0.0002488163881935179,0.07155339419841766,399.0,0,2025-11-07 06:50:03.851063
0.06345387548208237,0.00023934464843478054,0.06345387548208237,449.0,0,2025-11-07 06:56:53.772986
0.14404672384262085,0.0006276334170252085,0.14404672384262085,499.0,0,2025-11-07 07:03:43.786299
0.11206784099340439,0.0007210308103822172,0.11206784099340439,549.0,0,2025-11-07 07:10:33.770749
0.12019862234592438,0.0004977438948117197,0.12019862234592438,599.0,0,2025-11-07 07:17:24.044739
0.14713869988918304,0.0006160717457532883,0.14713869988918304,649.0,0,2025-11-07 07:24:14.265115
0.06740469485521317,0.0002478937094565481,0.06740469485521317,699.0,0,2025-11-07 07:31:04.399883
0.17105358839035034,0.0010129869915544987,0.17105358839035034,749.0,0,2025-11-07 07:37:54.942548
0.0800703912973404,0.0002778678899630904,0.0800703912973404,799.0,0,2025-11-07 07:44:44.956976
0.22667619585990906,0.0020731911063194275,0.22667619585990906,849.0,0,2025-11-07 07:51:34.881198
0.12335909903049469,0.001635329332202673,0.12335909903049469,899.0,0,2025-11-07 07:58:24.824366
0.13641229271888733,0.0005820619990117848,0.13641229271888733,949.0,0,2025-11-07 08:05:14.752665
0.04360431060194969,0.0001555299968458712,0.04360431060194969,999.0,0,2025-11-07 08:12:04.813715
0.14533722400665283,0.0008209170773625374,0.14533722400665283,1049.0,0,2025-11-07 08:18:54.764682
0.17020699381828308,0.0015672513982281089,0.17020699381828308,1099.0,0,2025-11-07 08:25:44.878204
0.1135483980178833,0.0007263791048899293,0.1135483980178833,1149.0,0,2025-11-07 08:32:34.936894
0.16013064980506897,0.00082513561937958,0.16013064980506897,1199.0,0,2025-11-07 08:39:24.798044
0.1042780876159668,0.0005099397385492921,0.1042780876159668,1249.0,0,2025-11-07 08:46:14.651963
0.21050581336021423,0.0034449275117367506,0.21050581336021423,1299.0,0,2025-11-07 08:53:04.780313
0.08077014982700348,0.00027821690309792757,0.08077014982700348,1349.0,0,2025-11-07 08:59:54.982233
0.08813931047916412,0.0004032023425679654,0.08813931047916412,1399.0,0,2025-11-07 09:06:45.131599
0.13852861523628235,0.0009442833252251148,0.13852861523628235,1449.0,0,2025-11-07 09:13:35.069566
0.0735173374414444,0.0002574293757788837,0.0735173374414444,1499.0,0,2025-11-07 09:20:47.621275
0.13102295994758606,0.0006024152971804142,0.13102295994758606,1549.0,0,2025-11-07 09:55:28.316552
0.0893625020980835,0.00035724518238566816,0.0893625020980835,1599.0,0,2025-11-07 10:02:17.766927
0.1264837682247162,0.0005941426497884095,0.1264837682247162,1649.0,0,2025-11-07 10:09:07.443280
0.2360188215970993,0.0033196182921528816,0.2360188215970993,1699.0,0,2025-11-07 10:15:57.195308
0.40757986903190613,0.03957106173038483,0.40757986903190613,1749.0,0,2025-11-07 10:22:46.729608
0.059670403599739075,0.00020686535572167486,0.059670403599739075,1799.0,0,2025-11-07 10:29:36.313280
0.2675992250442505,0.004967996384948492,0.2675992250442505,1849.0,0,2025-11-07 10:36:25.967824
0.13420328497886658,0.0005202709580771625,0.13420328497886658,1899.0,0,2025-11-07 10:43:15.667808
0.11876165121793747,0.0046053482219576836,0.11876165121793747,1949.0,0,2025-11-07 10:50:05.280139
0.07702823728322983,0.0002815905900206417,0.07702823728322983,1999.0,0,2025-11-07 10:56:55.100987
0.3279401957988739,0.0041978005319833755,0.3279401957988739,2049.0,0,2025-11-07 11:03:44.783742
0.09002681821584702,0.0005462608532980084,0.09002681821584702,2099.0,0,2025-11-07 11:10:34.265550
0.047717638313770294,0.00017055260832421482,0.047717638313770294,2149.0,0,2025-11-07 11:17:23.905393
0.1339346170425415,0.002061002654954791,0.1339346170425415,2199.0,0,2025-11-07 11:24:13.591040
0.2163275182247162,0.0010844864882528782,0.2163275182247162,2249.0,0,2025-11-07 11:31:03.172866
0.10397343337535858,0.00037731078919023275,0.10397343337535858,2299.0,0,2025-11-07 11:37:52.846567
0.12340980023145676,0.0004707850457634777,0.12340980023145676,2349.0,0,2025-11-07 11:44:42.456960
0.20107345283031464,0.0018489975482225418,0.20107345283031464,2399.0,0,2025-11-07 11:51:32.056443
0.10080485045909882,0.0003995543229393661,0.10080485045909882,2449.0,0,2025-11-07 11:58:21.806966
0.07403445243835449,0.0004102564125787467,0.07403445243835449,2499.0,0,2025-11-07 12:05:11.306898
0.04758354276418686,0.00020527694141492248,0.04758354276418686,2549.0,0,2025-11-07 12:12:01.110949
0.26019856333732605,0.006266983225941658,0.26019856333732605,2599.0,0,2025-11-07 12:18:50.897152
0.07928989827632904,0.0003213032032363117,0.07928989827632904,2649.0,0,2025-11-07 12:25:40.361476
0.08167669177055359,0.00033950069337151945,0.08167669177055359,2699.0,0,2025-11-07 12:32:30.083406
0.12661190330982208,0.0009067708160728216,0.12661190330982208,2749.0,0,2025-11-07 12:39:20.247792
0.07417937368154526,0.00027399056125432253,0.07417937368154526,2799.0,0,2025-11-07 12:46:10.075356
0.2565799951553345,0.009040337055921555,0.2565799951553345,2849.0,0,2025-11-07 12:52:59.460712
0.18894660472869873,0.005366226192563772,0.18894660472869873,2899.0,0,2025-11-07 12:59:49.052829
0.03070664033293724,0.00011124929005745798,0.03070664033293724,2949.0,0,2025-11-07 13:06:38.739401
0.05353289470076561,0.00019365045591257513,0.05353289470076561,2999.0,0,2025-11-07 13:13:47.971505
0.08127011358737946,0.0005316287279129028,0.08127011358737946,3049.0,0,2025-11-07 13:48:21.574918
0.138301283121109,0.0006474154652096331,0.138301283121109,3099.0,0,2025-11-07 13:55:10.905722
0.029935777187347412,0.00011087231541750953,0.029935777187347412,3149.0,0,2025-11-07 14:02:00.349531
0.12147270143032074,0.0004740321892313659,0.12147270143032074,3199.0,0,2025-11-07 14:08:49.688106
0.09411966800689697,0.0003769545292016119,0.09411966800689697,3249.0,0,2025-11-07 14:15:38.837187
0.058853887021541595,0.00020506291184574366,0.058853887021541595,3299.0,0,2025-11-07 14:22:28.186543
0.20748579502105713,0.001593392575159669,0.20748579502105713,3349.0,0,2025-11-07 14:29:17.676806
0.12053538858890533,0.0005773078883066773,0.12053538858890533,3399.0,0,2025-11-07 14:36:07.047555
0.16353946924209595,0.0009821749990805984,0.16353946924209595,3449.0,0,2025-11-07 14:42:56.380959
0.22428743541240692,0.005200668703764677,0.22428743541240692,3499.0,0,2025-11-07 14:49:46.173538
0.21036194264888763,0.0799005851149559,0.21036194264888763,3549.0,0,2025-11-07 14:56:35.586179
0.15972262620925903,0.0010552062885835767,0.15972262620925903,3599.0,0,2025-11-07 15:03:25.244468
0.07506482303142548,0.00033878497197292745,0.07506482303142548,3649.0,0,2025-11-07 15:10:14.858021
0.17174357175827026,0.0007673035142943263,0.17174357175827026,3699.0,0,2025-11-07 15:17:04.546858
0.12717080116271973,0.0006472798995673656,0.12717080116271973,3749.0,0,2025-11-07 15:23:54.187923
0.07495216280221939,0.0002986610634252429,0.07495216280221939,3799.0,0,2025-11-07 15:30:43.652162
0.2393846958875656,0.008072061464190483,0.2393846958875656,3849.0,0,2025-11-07 15:37:33.122951
0.1476171612739563,0.000847179617267102,0.1476171612739563,3899.0,0,2025-11-07 15:44:22.577225
0.15183554589748383,0.0010809821542352438,0.15183554589748383,3949.0,0,2025-11-07 15:51:11.799963
0.0597618967294693,0.0005647685029543936,0.0597618967294693,3999.0,0,2025-11-07 15:58:01.067926
0.17678801715373993,0.0025197183713316917,0.17678801715373993,4049.0,0,2025-11-07 16:04:50.430107
0.05736900493502617,0.00020722653425764292,0.05736900493502617,4099.0,0,2025-11-07 16:11:39.637185
0.09565054625272751,0.0004294906393624842,0.09565054625272751,4149.0,0,2025-11-07 16:18:28.885771
0.23071420192718506,0.0036755059845745564,0.23071420192718506,4199.0,0,2025-11-07 16:25:18.175830
0.2413683384656906,0.0063743507489562035,0.2413683384656906,4249.0,0,2025-11-07 16:32:07.364974
0.08362281322479248,0.00041367358062416315,0.08362281322479248,4299.0,0,2025-11-07 16:38:56.619476
0.06932064145803452,0.0002628997899591923,0.06932064145803452,4349.0,0,2025-11-07 16:45:46.108046
0.18747703731060028,0.004927619826048613,0.18747703731060028,4399.0,0,2025-11-07 16:52:35.569859
0.07947205007076263,0.00027208897517994046,0.07947205007076263,4449.0,0,2025-11-07 16:59:25.006144
0.18358290195465088,0.0032710744999349117,0.18358290195465088,4499.0,0,2025-11-07 17:06:37.234028
0.10804831981658936,0.0003872296947520226,0.10804831981658936,4549.0,0,2025-11-07 17:41:09.630339
Hello, thank you for your excellent open-source work. I am trying to reproduce your code using
python main.py --train --base configs/stableSRNew/v2-finetune_text_T_512.yaml --gpus GPU_ID, --name NAME --scale_lr False.Here's my training log. It's running on 6 A30 GPUs with max steps set to 100,000. Based on calculations, training will likely take about 10 days, which far exceeds the training time for stable_000117 you mentioned in #36. What issues might I have overlooked?
train/loss_simple_step,train/loss_vlb_step,train/loss_step,global_step,epoch,created_at
0.25144094228744507,0.001727868104353547,0.25144094228744507,49.0,0,2025-11-07 06:02:10.484610
0.0960497185587883,0.0004331176169216633,0.0960497185587883,99.0,0,2025-11-07 06:09:01.771131
0.10094963759183884,0.00045272731222212315,0.10094963759183884,149.0,0,2025-11-07 06:15:52.570149
0.079338937997818,0.0004949959111399949,0.079338937997818,199.0,0,2025-11-07 06:22:42.938209
0.09198877215385437,0.0003369307960383594,0.09198877215385437,249.0,0,2025-11-07 06:29:33.508085
0.16970014572143555,0.002384641207754612,0.16970014572143555,299.0,0,2025-11-07 06:36:23.734713
0.07759910821914673,0.0002961623831652105,0.07759910821914673,349.0,0,2025-11-07 06:43:13.754978
0.07155339419841766,0.0002488163881935179,0.07155339419841766,399.0,0,2025-11-07 06:50:03.851063
0.06345387548208237,0.00023934464843478054,0.06345387548208237,449.0,0,2025-11-07 06:56:53.772986
0.14404672384262085,0.0006276334170252085,0.14404672384262085,499.0,0,2025-11-07 07:03:43.786299
0.11206784099340439,0.0007210308103822172,0.11206784099340439,549.0,0,2025-11-07 07:10:33.770749
0.12019862234592438,0.0004977438948117197,0.12019862234592438,599.0,0,2025-11-07 07:17:24.044739
0.14713869988918304,0.0006160717457532883,0.14713869988918304,649.0,0,2025-11-07 07:24:14.265115
0.06740469485521317,0.0002478937094565481,0.06740469485521317,699.0,0,2025-11-07 07:31:04.399883
0.17105358839035034,0.0010129869915544987,0.17105358839035034,749.0,0,2025-11-07 07:37:54.942548
0.0800703912973404,0.0002778678899630904,0.0800703912973404,799.0,0,2025-11-07 07:44:44.956976
0.22667619585990906,0.0020731911063194275,0.22667619585990906,849.0,0,2025-11-07 07:51:34.881198
0.12335909903049469,0.001635329332202673,0.12335909903049469,899.0,0,2025-11-07 07:58:24.824366
0.13641229271888733,0.0005820619990117848,0.13641229271888733,949.0,0,2025-11-07 08:05:14.752665
0.04360431060194969,0.0001555299968458712,0.04360431060194969,999.0,0,2025-11-07 08:12:04.813715
0.14533722400665283,0.0008209170773625374,0.14533722400665283,1049.0,0,2025-11-07 08:18:54.764682
0.17020699381828308,0.0015672513982281089,0.17020699381828308,1099.0,0,2025-11-07 08:25:44.878204
0.1135483980178833,0.0007263791048899293,0.1135483980178833,1149.0,0,2025-11-07 08:32:34.936894
0.16013064980506897,0.00082513561937958,0.16013064980506897,1199.0,0,2025-11-07 08:39:24.798044
0.1042780876159668,0.0005099397385492921,0.1042780876159668,1249.0,0,2025-11-07 08:46:14.651963
0.21050581336021423,0.0034449275117367506,0.21050581336021423,1299.0,0,2025-11-07 08:53:04.780313
0.08077014982700348,0.00027821690309792757,0.08077014982700348,1349.0,0,2025-11-07 08:59:54.982233
0.08813931047916412,0.0004032023425679654,0.08813931047916412,1399.0,0,2025-11-07 09:06:45.131599
0.13852861523628235,0.0009442833252251148,0.13852861523628235,1449.0,0,2025-11-07 09:13:35.069566
0.0735173374414444,0.0002574293757788837,0.0735173374414444,1499.0,0,2025-11-07 09:20:47.621275
0.13102295994758606,0.0006024152971804142,0.13102295994758606,1549.0,0,2025-11-07 09:55:28.316552
0.0893625020980835,0.00035724518238566816,0.0893625020980835,1599.0,0,2025-11-07 10:02:17.766927
0.1264837682247162,0.0005941426497884095,0.1264837682247162,1649.0,0,2025-11-07 10:09:07.443280
0.2360188215970993,0.0033196182921528816,0.2360188215970993,1699.0,0,2025-11-07 10:15:57.195308
0.40757986903190613,0.03957106173038483,0.40757986903190613,1749.0,0,2025-11-07 10:22:46.729608
0.059670403599739075,0.00020686535572167486,0.059670403599739075,1799.0,0,2025-11-07 10:29:36.313280
0.2675992250442505,0.004967996384948492,0.2675992250442505,1849.0,0,2025-11-07 10:36:25.967824
0.13420328497886658,0.0005202709580771625,0.13420328497886658,1899.0,0,2025-11-07 10:43:15.667808
0.11876165121793747,0.0046053482219576836,0.11876165121793747,1949.0,0,2025-11-07 10:50:05.280139
0.07702823728322983,0.0002815905900206417,0.07702823728322983,1999.0,0,2025-11-07 10:56:55.100987
0.3279401957988739,0.0041978005319833755,0.3279401957988739,2049.0,0,2025-11-07 11:03:44.783742
0.09002681821584702,0.0005462608532980084,0.09002681821584702,2099.0,0,2025-11-07 11:10:34.265550
0.047717638313770294,0.00017055260832421482,0.047717638313770294,2149.0,0,2025-11-07 11:17:23.905393
0.1339346170425415,0.002061002654954791,0.1339346170425415,2199.0,0,2025-11-07 11:24:13.591040
0.2163275182247162,0.0010844864882528782,0.2163275182247162,2249.0,0,2025-11-07 11:31:03.172866
0.10397343337535858,0.00037731078919023275,0.10397343337535858,2299.0,0,2025-11-07 11:37:52.846567
0.12340980023145676,0.0004707850457634777,0.12340980023145676,2349.0,0,2025-11-07 11:44:42.456960
0.20107345283031464,0.0018489975482225418,0.20107345283031464,2399.0,0,2025-11-07 11:51:32.056443
0.10080485045909882,0.0003995543229393661,0.10080485045909882,2449.0,0,2025-11-07 11:58:21.806966
0.07403445243835449,0.0004102564125787467,0.07403445243835449,2499.0,0,2025-11-07 12:05:11.306898
0.04758354276418686,0.00020527694141492248,0.04758354276418686,2549.0,0,2025-11-07 12:12:01.110949
0.26019856333732605,0.006266983225941658,0.26019856333732605,2599.0,0,2025-11-07 12:18:50.897152
0.07928989827632904,0.0003213032032363117,0.07928989827632904,2649.0,0,2025-11-07 12:25:40.361476
0.08167669177055359,0.00033950069337151945,0.08167669177055359,2699.0,0,2025-11-07 12:32:30.083406
0.12661190330982208,0.0009067708160728216,0.12661190330982208,2749.0,0,2025-11-07 12:39:20.247792
0.07417937368154526,0.00027399056125432253,0.07417937368154526,2799.0,0,2025-11-07 12:46:10.075356
0.2565799951553345,0.009040337055921555,0.2565799951553345,2849.0,0,2025-11-07 12:52:59.460712
0.18894660472869873,0.005366226192563772,0.18894660472869873,2899.0,0,2025-11-07 12:59:49.052829
0.03070664033293724,0.00011124929005745798,0.03070664033293724,2949.0,0,2025-11-07 13:06:38.739401
0.05353289470076561,0.00019365045591257513,0.05353289470076561,2999.0,0,2025-11-07 13:13:47.971505
0.08127011358737946,0.0005316287279129028,0.08127011358737946,3049.0,0,2025-11-07 13:48:21.574918
0.138301283121109,0.0006474154652096331,0.138301283121109,3099.0,0,2025-11-07 13:55:10.905722
0.029935777187347412,0.00011087231541750953,0.029935777187347412,3149.0,0,2025-11-07 14:02:00.349531
0.12147270143032074,0.0004740321892313659,0.12147270143032074,3199.0,0,2025-11-07 14:08:49.688106
0.09411966800689697,0.0003769545292016119,0.09411966800689697,3249.0,0,2025-11-07 14:15:38.837187
0.058853887021541595,0.00020506291184574366,0.058853887021541595,3299.0,0,2025-11-07 14:22:28.186543
0.20748579502105713,0.001593392575159669,0.20748579502105713,3349.0,0,2025-11-07 14:29:17.676806
0.12053538858890533,0.0005773078883066773,0.12053538858890533,3399.0,0,2025-11-07 14:36:07.047555
0.16353946924209595,0.0009821749990805984,0.16353946924209595,3449.0,0,2025-11-07 14:42:56.380959
0.22428743541240692,0.005200668703764677,0.22428743541240692,3499.0,0,2025-11-07 14:49:46.173538
0.21036194264888763,0.0799005851149559,0.21036194264888763,3549.0,0,2025-11-07 14:56:35.586179
0.15972262620925903,0.0010552062885835767,0.15972262620925903,3599.0,0,2025-11-07 15:03:25.244468
0.07506482303142548,0.00033878497197292745,0.07506482303142548,3649.0,0,2025-11-07 15:10:14.858021
0.17174357175827026,0.0007673035142943263,0.17174357175827026,3699.0,0,2025-11-07 15:17:04.546858
0.12717080116271973,0.0006472798995673656,0.12717080116271973,3749.0,0,2025-11-07 15:23:54.187923
0.07495216280221939,0.0002986610634252429,0.07495216280221939,3799.0,0,2025-11-07 15:30:43.652162
0.2393846958875656,0.008072061464190483,0.2393846958875656,3849.0,0,2025-11-07 15:37:33.122951
0.1476171612739563,0.000847179617267102,0.1476171612739563,3899.0,0,2025-11-07 15:44:22.577225
0.15183554589748383,0.0010809821542352438,0.15183554589748383,3949.0,0,2025-11-07 15:51:11.799963
0.0597618967294693,0.0005647685029543936,0.0597618967294693,3999.0,0,2025-11-07 15:58:01.067926
0.17678801715373993,0.0025197183713316917,0.17678801715373993,4049.0,0,2025-11-07 16:04:50.430107
0.05736900493502617,0.00020722653425764292,0.05736900493502617,4099.0,0,2025-11-07 16:11:39.637185
0.09565054625272751,0.0004294906393624842,0.09565054625272751,4149.0,0,2025-11-07 16:18:28.885771
0.23071420192718506,0.0036755059845745564,0.23071420192718506,4199.0,0,2025-11-07 16:25:18.175830
0.2413683384656906,0.0063743507489562035,0.2413683384656906,4249.0,0,2025-11-07 16:32:07.364974
0.08362281322479248,0.00041367358062416315,0.08362281322479248,4299.0,0,2025-11-07 16:38:56.619476
0.06932064145803452,0.0002628997899591923,0.06932064145803452,4349.0,0,2025-11-07 16:45:46.108046
0.18747703731060028,0.004927619826048613,0.18747703731060028,4399.0,0,2025-11-07 16:52:35.569859
0.07947205007076263,0.00027208897517994046,0.07947205007076263,4449.0,0,2025-11-07 16:59:25.006144
0.18358290195465088,0.0032710744999349117,0.18358290195465088,4499.0,0,2025-11-07 17:06:37.234028
0.10804831981658936,0.0003872296947520226,0.10804831981658936,4549.0,0,2025-11-07 17:41:09.630339