Yanshuai Cao 曹颜帅

Selected Publications:

* denotes equal contribution

1. ACL

   Code Generation from Natural Language with Less Prior Knowledge and More Monolingual Data

   Sajad Norouzi, Keyi Tang, and Yanshuai Cao

   In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics 2021

   Abs Bibtex HTML Code

   Training datasets for semantic parsing are typically small due to the higher expertise required for annotation than most other NLP tasks. As a result, models for this application usually need additional prior knowledge to be built into the architecture or algorithm. The increased dependency on human experts hinders automation and raises the development and maintenance costs in practice. This work investigates whether a generic transformer-based seq2seq model can achieve competitive performance with minimal code-generation-specific inductive bias design. By exploiting a relatively sizeable monolingual corpus of the target programming language, which is cheap to mine from the web, we achieved 81.03% exact match accuracy on Django and 32.57 BLEU score on CoNaLa. Both are SOTA to the best of our knowledge. This positive evidence highlights a potentially easier path toward building accurate semantic parsers in practice.

    @inproceedings{norouzi-etal-2021-code,
     abbr = {ACL},
     title = {Code Generation from Natural Language with Less Prior Knowledge and More Monolingual Data},
     author = {Norouzi, Sajad and Tang, Keyi and Cao, Yanshuai},
     booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics},
     month = aug,
     year = {2021},
     address = {Online},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2021.acl-short.98},
     doi = {10.18653/v1/2021.acl-short.98},
     pages = {776--785},
     code = {https://github.com/BorealisAI/code-gen-TAE},
     selected = {true},
     html = {https://aclanthology.org/2021.acl-short.98/}
   }
    

2. ACL

   Optimizing Deeper Transformers on Small Datasets

   Peng Xu, Dhruv Kumar, Wei Yang, Wenjie Zi, Keyi Tang, Chenyang Huang, Jackie Chi Kit Cheung, Simon J.D. Prince, and Yanshuai Cao

   In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics 2021

   Abs Bibtex HTML Code

   It is a common belief that training deep transformers from scratch requires large datasets. Consequently, for small datasets, people usually use shallow and simple additional layers on top of pre-trained models during fine-tuning. This work shows that this does not always need to be the case: with proper initialization and optimization, the benefits of very deep transformers can carry over to challenging tasks with small datasets, including Text-to-SQL semantic parsing and logical reading comprehension. In particular, we successfully train 48 layers of transformers, comprising 24 fine-tuned layers from pre-trained RoBERTa and 24 relation-aware layers trained from scratch. With fewer training steps and no task-specific pre-training, we obtain the state of the art performance on the challenging cross-domain Text-to-SQL parsing benchmark Spider. We achieve this by deriving a novel Data dependent Transformer Fixed-update initialization scheme (DT-Fixup), inspired by the prior T-Fixup work. Further error analysis shows that increasing depth can help improve generalization on small datasets for hard cases that require reasoning and structural understanding.

    @inproceedings{xu-etal-2021-optimizing,
     abbr = {ACL},
     title = {Optimizing Deeper Transformers on Small Datasets},
     author = {Xu, Peng and Kumar, Dhruv and Yang, Wei and Zi, Wenjie and Tang, Keyi and Huang, Chenyang and Cheung, Jackie Chi Kit and Prince, Simon J.D. and Cao, Yanshuai},
     booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics},
     month = aug,
     year = {2021},
     address = {Online},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2021.acl-long.163},
     doi = {10.18653/v1/2021.acl-long.163},
     pages = {2089--2102},
     html = {https://aclanthology.org/2021.acl-long.163},
     code = {https://github.com/BorealisAI/DT-Fixup},
     selected = {true}
   }
    

3. AISTATS

   Better Long-Range Dependency By Bootstrapping A Mutual Information Regularizer

   Yanshuai Cao*, and Peng Xu*

   In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics 2020

   Abs Bibtex arXiv HTML PDF Code

   In this work, we develop a novel regularizer to improve the learning of long-range dependency of sequence data. Applied on language modelling, our regularizer expresses the inductive bias that sequence variables should have high mutual information even though the model might not see abundant observations for complex long-range dependency. We show how the “next sentence prediction (classification)" heuristic can be derived in a principled way from our mutual information estimation framework, and be further extended to maximize the mutual information of sequence variables. The proposed approach not only is effective at increasing the mutual information of segments under the learned model but more importantly, leads to a higher likelihood on holdout data, and improved generation quality. Code is releasedat https://github.com/BorealisAI/BMI.

    @inproceedings{pmlr-v108-cao20a,
     abbr = {AISTATS},
     title = {Better Long-Range Dependency By Bootstrapping A Mutual Information Regularizer},
     author = {Cao, Yanshuai and Xu, Peng},
     booktitle = {Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics},
     pages = {3991--4001},
     year = {2020},
     editor = {Chiappa, Silvia and Calandra, Roberto},
     volume = {108},
     series = {Proceedings of Machine Learning Research},
     address = {Online},
     month = {26--28 Aug},
     publisher = {PMLR},
     pdf = {http://proceedings.mlr.press/v108/cao20a/cao20a.pdf},
     html = {http://proceedings.mlr.press/v108/cao20a.html},
     arxiv = {1905.11978},
     code = {https://github.com/BorealisAI/BMI},
     selected = {true},
     equal_last = {Cao, Xu}
   }
    

4. ICML

   Evaluating Lossy Compression Rates of Deep Generative Models

   Sicong Huang*, Alireza Makhzani*, Yanshuai Cao, and Roger Grosse

   In Proceedings of the 37th International Conference on Machine Learning 2020

   Abs Bibtex arXiv HTML PDF Code

   The field of deep generative modeling has succeeded in producing astonishingly realistic-seeming images and audio, but quantitative evaluation remains a challenge. Log-likelihood is an appealing metric due to its grounding in statistics and information theory, but it can be challenging to estimate for implicit generative models, and scalar-valued metrics give an incomplete picture of a model’s quality. In this work, we propose to use rate distortion (RD) curves to evaluate and compare deep generative models. While estimating RD curves is seemingly even more computationally demanding than log-likelihood estimation, we show that we can approximate the entire RD curve using nearly the same computations as were previously used to achieve a single log-likelihood estimate. We evaluate lossy compression rates of VAEs, GANs, and adversarial autoencoders (AAEs) on the MNIST and CIFAR10 datasets. Measuring the entire RD curve gives a more complete picture than scalar-valued metrics, and we arrive at a number of insights not obtainable from log-likelihoods alone.

    @inproceedings{pmlr-v119-huang20c,
     abbr = {ICML},
     title = {Evaluating Lossy Compression Rates of Deep Generative Models},
     author = {Huang, Sicong and Makhzani, Alireza and Cao, Yanshuai and Grosse, Roger},
     booktitle = {Proceedings of the 37th International Conference on Machine Learning},
     pages = {4444--4454},
     year = {2020},
     editor = {III, Hal Daumé and Singh, Aarti},
     volume = {119},
     series = {Proceedings of Machine Learning Research},
     address = {Virtual},
     month = {13--18 Jul},
     publisher = {PMLR},
     pdf = {http://proceedings.mlr.press/v119/huang20c/huang20c.pdf},
     html = {http://proceedings.mlr.press/v119/huang20c.html},
     arxiv = {2008.06653},
     code = {https://github.com/huangsicong/rate_distortion},
     selected = {true},
     equal_last = {Huang, Makhzani}
   }
    

5. ICML

   On Variational Learning of Controllable Representations for Text without Supervision

   Peng Xu, Jackie Chi Kit Cheung, and Yanshuai Cao

   In Proceedings of the 37th International Conference on Machine Learning 2020

   Abs Bibtex arXiv HTML PDF Code

   The variational autoencoder (VAE) can learn the manifold of natural images on certain datasets, as evidenced by meaningful interpolating or extrapolating in the continuous latent space. However, on discrete data such as text, it is unclear if unsupervised learning can discover similar latent space that allows controllable manipulation. In this work, we find that sequence VAEs trained on text fail to properly decode when the latent codes are manipulated, because the modified codes often land in holes or vacant regions in the aggregated posterior latent space, where the decoding network fails to generalize. Both as a validation of the explanation and as a fix to the problem, we propose to constrain the posterior mean to a learned probability simplex, and performs manipulation within this simplex. Our proposed method mitigates the latent vacancy problem and achieves the first success in unsupervised learning of controllable representations for text. Empirically, our method outperforms unsupervised baselines and strong supervised approaches on text style transfer, and is capable of performing more flexible fine-grained control over text generation than existing methods.

    @inproceedings{pmlr-v119-xu20a,
     abbr = {ICML},
     title = {On Variational Learning of Controllable Representations for Text without Supervision},
     author = {Xu, Peng and Cheung, Jackie Chi Kit and Cao, Yanshuai},
     booktitle = {Proceedings of the 37th International Conference on Machine Learning},
     pages = {10534--10543},
     year = {2020},
     editor = {III, Hal Daumé and Singh, Aarti},
     volume = {119},
     series = {Proceedings of Machine Learning Research},
     address = {Virtual},
     month = {13--18 Jul},
     publisher = {PMLR},
     pdf = {http://proceedings.mlr.press/v119/xu20a/xu20a.pdf},
     url = {http://proceedings.mlr.press/v119/xu20a.html},
     html = {http://proceedings.mlr.press/v119/xu20a.html},
     arxiv = {1905.11975},
     code = {https://github.com/BorealisAI/CP-VAE},
     selected = {true}
   }
    

6. ICLR

   Improving GAN Training via Binarized Representation Entropy (BRE) Regularization

   Yanshuai Cao, Gavin Weiguang Ding, Kry Yik-Chau Lui, and Ruitong Huang

   International Conference on Learning Representations, ICLR 2018

   Abs Bibtex arXiv HTML Code

   We propose a novel regularizer to improve the training of Generative Adversarial Networks (GANs). The motivation is that when the discriminator D spreads out its model capacity in the right way, the learning signals given to the generator G are more informative and diverse. These in turn help G to explore better and discover the real data manifold while avoiding large unstable jumps due to the erroneous extrapolation made by D. Our regularizer guides the rectifier discriminator D to better allocate its model capacity, by encouraging the binary activation patterns on selected internal layers of D to have a high joint entropy. Experimental results on both synthetic data and real datasets demonstrate improvements in stability and convergence speed of the GAN training, as well as higher sample quality. The approach also leads to higher classification accuracies in semi-supervised learning.

    @article{Cao2018Improving,
     abbr = {ICLR},
     title = {Improving GAN Training via Binarized Representation Entropy (BRE) Regularization},
     author = {Cao, Yanshuai and Ding, Gavin Weiguang and Lui, Kry Yik-Chau and Huang, Ruitong},
     journal = {International Conference on Learning Representations, {ICLR}},
     year = {2018},
     html = {https://openreview.net/forum?id=BkLhaGZRW},
     arxiv = {1805.03644},
     code = {https://github.com/BorealisAI/bre-gan},
     selected = {true}
   }
    

7. ACL

   Adversarial Contrastive Estimation

   Avishek Joey Bose*, Huan Ling*, and Yanshuai Cao*

   In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) 2018

   Abs Bibtex arXiv HTML

   Learning by contrasting positive and negative samples is a general strategy adopted by many methods. Noise contrastive estimation (NCE) for word embeddings and translating embeddings for knowledge graphs are examples in NLP employing this approach. In this work, we view contrastive learning as an abstraction of all such methods and augment the negative sampler into a mixture distribution containing an adversarially learned sampler. The resulting adaptive sampler finds harder negative examples, which forces the main model to learn a better representation of the data. We evaluate our proposal on learning word embeddings, order embeddings and knowledge graph embeddings and observe both faster convergence and improved results on multiple metrics.

    @inproceedings{bose-etal-2018-adversarial,
     abbr = {ACL},
     title = {Adversarial Contrastive Estimation},
     author = {Bose, Avishek Joey and Ling, Huan and Cao, Yanshuai},
     booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
     month = jul,
     year = {2018},
     address = {Melbourne, Australia},
     publisher = {Association for Computational Linguistics},
     url = {https://www.aclweb.org/anthology/P18-1094},
     doi = {10.18653/v1/P18-1094},
     pages = {1021--1032},
     arxiv = {1805.03642},
     html = {https://www.aclweb.org/anthology/P18-1094/},
     selected = {true},
     equal_last = {Bose, Ling, Cao}
   }