Dear Author,
We have noticed that in the SFNET code, you have used semSQL's intermediate representation as the evaluation benchmark instead of the final SQL, which is not the ultimate goal of the semantic parsing task. Therefore, we attempted to follow the method provided in IRNET to convert semSQL into formal SQL, we obtained the following results on the SQL Exact Match metric:
ACC_a = 54.67; ACC_w = 55.78; BWT = -0.46; FWT=41.09
Although it is normal for the results to vary, we observed significant changes in the model's performance on the BWT and FWT metrics (originally BWT = -1.0; FWT = 45.9). Notably, the model's performance on catastrophic forgetting almost disappeared. Could you explain why this is the case?
Dear Author,
We have noticed that in the SFNET code, you have used semSQL's intermediate representation as the evaluation benchmark instead of the final SQL, which is not the ultimate goal of the semantic parsing task. Therefore, we attempted to follow the method provided in IRNET to convert semSQL into formal SQL, we obtained the following results on the SQL Exact Match metric:
ACC_a = 54.67; ACC_w = 55.78; BWT = -0.46; FWT=41.09
Although it is normal for the results to vary, we observed significant changes in the model's performance on the BWT and FWT metrics (originally BWT = -1.0; FWT = 45.9). Notably, the model's performance on catastrophic forgetting almost disappeared. Could you explain why this is the case?