Today’s deep dive into the latest research from Arxiv reveals a controversy brewing over Google’s reinforcement learning for chip macro placement, a breakthrough in extending the context window of large language models, and the promising application of deep learning in automated database tuning. We’ll be taking a closer look at how transformers fare with arithmetic and longer sequences, and the surprising findings on the planning abilities of large language models. As always, we’ll be juxtaposing these insights with the candid discussions from Hacker News, delving into the issues of reproducibility, the impact of flawed research, and the ethical standards of academic research. Buckle up for a riveting exploration of the frontiers of AI research.

Top Papers

1) The False Dawn Reevaluating Googles Reinforcement Learning for Chip Macro Placement

Summary:

Google’s reinforcement learning approach for chip macro placement is being criticized for lack of transparency and incomplete source code, with concerns about the study’s integrity and reproducibility leading to an investigation by Nature editors.

View PDF | Chat with this paper

• Google’s reinforcement learning (RL) approach for chip macro placement, as described in a 2021 Nature paper, has come under scrutiny due to poorly documented claims and omissions in the methodology.
• Two separate evaluations have shown that Google’s RL lags behind human designers, Simulated Annealing, and commercial software.
• The integrity of the Nature paper is called into question due to errors in conduct, analysis, and reporting.
• The controversy has been covered by media outlets, highlighting allegations of fraud and scientific misconduct.
• The methodology used in the Nature paper had notable shortcomings, including the use of proprietary Google TPU circuit design blocks and a simplified proxy cost function.
• The baselines used in the paper did not outperform other methods such as Simulated Annealing and commercial EDA tools.
• The study faced reproducibility issues and objections from researchers who attempted to replicate the results.
• The original paper on Google’s reinforcement learning for chip macro placement has been called into question due to issues with reproducibility, misleading comparisons, and barriers to improvement.

Hacker News:

The article critiques Google’s RL for Chip Macro Placement methodology, highlighting its lack of reproducibility and inferior performance, while also discussing the consequences of flawed research, the role of tenure in protecting researchers, and the ethical standards of academics. View on HN

• The author criticizes Google’s approach to publishing research on chip macro placement, stating that the lack of details made it impossible to reproduce the results.
• Other mechanistic approaches to chip placement were found to perform better than Google’s RL approach.
• The author questions why Google did not allow the publication of a coauthored paper that found major flaws in their approach, despite its results being corroborated by another published paper.
• The critique reflects poorly on both Google and the journal Nature for accepting the publication without sufficient rigor.
• The author suggests that if the authors were academics with tenure, their careers would be severely impacted by this controversy.

2) Extending Context Window of Large Language Models

Summary:

The document discusses how Position Interpolation can extend the context window of large language models up to 32768 tokens, outperforming direct fine-tuning and maintaining model architecture, with evaluation on benchmark tasks and long document summarization.

View PDF | Chat with this paper

• Position Interpolation (PI) is a method to extend the context window sizes of large language models (LLMs) up to 32768 tokens with minimal fine-tuning.
• PI down-scales the input position indices to match the original context window size, avoiding high attention scores that disrupt the self-attention mechanism.
• Extending the context window of LLMs is necessary for tasks like long conversations, summarizing long documents, and long-term planning.
• Fine-tuning existing pre-trained Transformer models with longer context windows is inefficient, while PI enables context window extensions for pre-trained LLMs.
• Empirical results show that Position Interpolation is highly effective and efficient, requiring only a short period of fine-tuning for the model to fully adapt to greatly extended context windows.

3) Utilizing Deep Learning for Automated Database Tuning

Summary:

This article presents an automated solution for managing database system configurations using machine learning techniques, including GMM clustering and ensemble models, to improve latency prediction and performance of automated DBMS tuning.

View PDF | Chat with this paper

• The article discusses the challenges of managing database system configurations and the lack of standardization among configuration knobs.
• The authors propose an automated solution that utilizes supervised and unsupervised machine learning techniques to identify influential knobs, analyze unseen workloads, and provide recommendations for optimal knob settings.
• The effectiveness of the proposed approach is demonstrated through the evaluation of a tool called OtterTune on three different database management systems (DBMSs).
• The authors extend the automated technique introduced in the original OtterTune paper by utilizing previously collected training data to optimize new DBMS deployments and improve latency prediction.
• The article highlights the complexity of DBMSs and the multitude of configuration knobs that impact performance and scalability, leading to the need for automatic tuning tools.
• The authors propose a new approach that reuses training data from previous tuning sessions to optimize DBMS performance for new applications, reducing the time and resources needed for optimization.
• The main objective of the authors’ work is to extend OtterTune and propose novel machine learning models from previously collected data to prune redundant metrics, map unseen workloads, and improve latency prediction.
• The experiment discussed in the document aimed to automate the tuning of database management system (DBMS) configurations using deep learning techniques.

4) Length Generalization in Arithmetic Transformers

Summary:

This text discusses how transformers struggle with arithmetic and longer sequences, but the use of relative position embeddings and train set priming can improve generalization, particularly in multiplication tasks.

View PDF | Chat with this paper

• Transformers struggle with simple tasks like integer arithmetic and generalizing to longer sequences.
• Relative position embeddings enable length generalization for addition tasks but fail for multiplication.
• Train set priming, by adding long sequences to the training set, improves models’ ability to generalize to larger multiplication examples.
• The number of priming examples required scales logarithmically with the number of training examples and linearly with the extrapolation length.
• Relative position embeddings perform better than absolute position embeddings in terms of length generalization.
• Priming the train set with 35-digit numbers allows for extrapolation to 35-digit operands, while priming on numbers from 6 to 35 enables extrapolation to all lengths up to 35.
• Model failures in addition tasks are observed in cases involving three or more carries and two consecutive carries.
• Open questions for future research include extending priming to other mathematical problems and exploring priming in natural language processing tasks.

5) Investigating Planning Abilities of Large Language Models

Summary:

Large language models (LLMs) have limited ability to generate executable plans autonomously, but they can serve as heuristic guidance for other agents in the logistics domain; caution is needed to verify correctness and bias, and safety and potential bias perpetuation should be carefully considered when using LLMs for planning.

View PDF | Chat with this paper

• Large language models (LLMs) were investigated for their planning abilities in generating plans autonomously and as heuristic guidance for other agents.
• LLMs had limited success in generating executable plans autonomously, with the best model (GPT-4) achieving an average success rate of around 12%.
• LLM-generated plans showed more promise in the heuristic mode, improving the search process for underlying planners and benefiting from external verifiers’ feedback.
• While LLMs performed poorly in autonomous planning, their generated plans could assist AI planners and be refined through backprompting.
• The study highlighted the limitations and potential benefits of using LLMs in planning tasks, emphasizing the importance of verification for correctness and bias.