Zongyu (Johnson) Lin「林宗裕」

I am a CS Ph.D Student at UCLA, co-advised by Prof. Yizhou Sun and Prof. Kaiwei Chang. Before coming to UCLA, I have spent almost a year in Moonshot.AI, as a LLM researcher. I was one of the major contributor of training large language models with extremely long context, achieving the state-of-the-art performance on many long context tasks compared with GPT4 and Claude2, and also participated in visual generation project. I completed my bachelor's degree of Electronic Enginnering at Tsinghua University. Luckily, I have worked with Prof. Zhilin Yang, Prof. Yong Li, Prof. Hanan Samet and Prof. Cyrus Shahabi.

My research interest lies broadly in natural language processing and general machine learning. I have done some work including self-training, instruction finetuning and zero-shot task generalization of LLMs. Most Recently, I am interested in (1) studying the self-evolution paradigm of large language models as well as (2) exploring scalable architectures and recipes for multi-modal generation. Feel free to contact me for casual chat or discussion if you are also interested in these topics.

Email: lzyxx17 [at] gmail.com

Google Scholar  

My research interest lies broadly in natural language processing and general machine learning. Most Recently, I am interested in

(1) improving the self-evolution of large language models;

(2) exploring scalable architectures and recipes for multi-modal generation;

(3) improve better physics understanding for vision / embodied language models

Generative modeling has been the dominant approach for large-scale pretraining and zero-shot generalization. In this work, we challenge this convention by showing that discriminative approaches perform substantially better than generative ones on a large number of NLP tasks. Technically, we train a single discriminator to predict whether a text sample comes from the true data distribution, similar to GANs. Since many NLP tasks can be formulated as selecting from a few options, we use this discriminator to predict the option with the highest probability. This simple formulation achieves state-of-the-art zero-shot results on the T0 benchmark, outperforming T0 by 16.0%, 7.8%, and 11.5% respectively on different scales. Meanwhile, our approach requires minimal prompting efforts, which largely improves robustness and is essential for real-world applications.

Recent work has achieved remarkable zero-shot performance with multi-task prompted pretraining, but little has been understood. For the first time, we show that training on a small number of key tasks beats using all the training tasks, while removing these key tasks substantially hurts performance. We also find that these key tasks are mostly question answering (QA) tasks. We design a shuffle experiment to further show that training on these QA tasks leads to better cross-task generalization in multi-task learning under various training/test task splits. These novel findings combined deepen our understanding about zero-generalization— training on certain tasks such as QA encodes general knowledge transferable to a wide range of tasks, which explains the improved zero-shot performance in recent progress. In addition, to automate this procedure, we devise a method to identify and upsample key training tasks without observing the test tasks based on cross validation. Empirically, our approach achieves improved results across various model scales and tasks.

Label noise is ubiquitous in various machine learning scenarios such as self-labeling with model predictions and erroneous data annotation. Many existing approaches are based on heuristics such as sample losses, which might not be flexible enough to achieve optimal solutions. Meta learning based methods address this issue by learning a data selection function, but can be hard to optimize. In light of these pros and cons, we propose SENT (Selection-Enhanced Noisy label Training) that does not rely on meta learning while having the flexibility of being data-driven. SENT transfers the noise distribution to a clean set and trains a model to distinguish noisy labels from clean ones using model-based features. Empirically, on a wide range of tasks including text classification and speech recognition, SENT improves performance over strong baselines under the settings of self-training and label corruption.

Label noise is ubiquitous in various machine learning scenarios such as self-labeling with model predictions and erroneous data annotation. Many existing approaches are based on heuristics such as sample losses, which might not be flexible enough to achieve optimal solutions. Meta learning based methods address this issue by learning a data selection function, but can be hard to optimize. In light of these pros and cons, we propose SENT (Selection-Enhanced Noisy label Training) that does not rely on meta learning while having the flexibility of being data-driven. SENT transfers the noise distribution to a clean set and trains a model to distinguish noisy labels from clean ones using model-based features. Empirically, on a wide range of tasks including text classification and speech recognition, SENT improves performance over strong baselines under the settings of self-training and label corruption.

We propose an end-to-end graph convolutional recurrent network called HAGEN with several novel designs for crime prediction. Specifically, our framework could jointly capture the crime correlation between regions and the temporal crime dynamics by combining an adaptive region graph learning module with the Diffusion Convolution Gated Recurrent Unit (DCGRU). Based on the homophily assumption of GNN (i.e., graph convolution works better where neighboring nodes share the same label), we propose a homophily-aware constraint to regularize the optimization of the region graph so that neighboring region nodes on the learned graph share similar crime patterns.

We propose a general system to recover vehicle trajectories at the level of the road intersection, where a novel iterative framework is developed to combine both vehicle clustering and trajectory recovery tasks.

We propose a DFA-based model which can generate interpretable features automatically from raw mobility data for fine-grained health sensing.

We propose a semantic-enhanced urban mobility embedding model for user profiling, and reveal meaningful patterns in all spatial, temporal and urban structure domains.

We introduce a new end-to-end spatiotemporal learning framework dubbed CrimeForecaster that: 1) represents the geographical extents of neighborhoods and their correlations in a graph; 2) uses graph convolution to predict crimes.

LLM Researcher, Moonshot.AI 2023

Quant Researcher Intern, Ubiquant, Top Hedge Fund in China. 2022

Research Intern, Sensetime, China, 2021

Comprehensive Outstanding Scholarship(~10/280), Tsinghua University. 2020

Excellent Technology Innovation Scholarship, Tsinghua University. 2020

First Prize in Software Design Contest, Department of Electronic Enginnering, Tsinghua University. 2018

Sports! I really enjoy playing ballgames like football and tennis. I am a big fan of Lionel Messi, Rafael Nadal and Stephen Curry! Also, I love running, swimming and hiking.