Archives
All the articles I've archived.
Nested Learning: The Illusion of Deep Learning Architectures
Updated: at 17:08Published: at 11:40谷歌新作，号称“深度学习新范式”。提到了异步，具体指的是让模型靠近输入的位置的更新频率高于靠后的位置，这个思路和之前Sakana AI的那个文章有点像。但文章里面的东西感觉全都是Fast Weight Programming的内容，arxiv的文章全文也一直没挂出来。
Kimi Linear: An Expressive, Efficient Attention Architecture
Updated: at 19:10Published: at 13:55Kimi Linear，有比较详细的实验&Scale Up。有Linear Attention可以去掉RoPE这个结论还是比较惊喜的。
Speed Always Wins: A Survey on Efficient Architectures for Large Language Models
Updated: at 15:15Published: at 17:05AI Lab关于”广义“LLM推理加速的工作，包括Linear Attention，Sparse Attention，Diffusion LLM，Applications等。
Neuromorphic Principles for Efficient Large Language Models on Intel Loihi 2
Updated: at 00:39Published: at 23:32ICLR2025 Workshop，基于HAQ实现的Matmul-Free SNN LLM（虽然只做了370M参数的实验）部署到Loihi2上，实现了相比于Qwen-500M 模型3\timesThroughput和2\times能效。但说实话文章内容关键点都没怎么讲，也没有什么特别很exciting的东西。
Parallelizing Linear Transformers with the Delta Rule over Sequence Length
Updated: at 16:46Published: at 14:43DeltaNet
Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity
Updated: at 15:07Published: at 13:50VLDB2024，阿里的工作，看起来工程特别扎实。LLM任务上只通过对weight做sparse load就能在decode阶段获得3-4倍的提速。
SpikingBrain-瞬息 1.0技术报告：原生国产自主可控类脑脉冲大模型
Updated: at 14:34Published: at 10:46李国齐老师组的新工作技术报告。说实话，我并不觉得这是一个正经的SNN-LLM工作，感觉已经完全是Linear Attention国产化的工作了。很难评价。
MLP Memory: Language Modeling with Retriever-pretrained External Memory
Published: at 14:22用MLP学习并代替RAG中kNN输出的概率分布。
Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
Published: at 16:04ACL2025 Best Paper，DeepSeek新作。分层KV Cache提高稀疏度，在训练和推理阶段同时提高性能。
GPU上的SNN稀疏加速
Updated: at 11:09Published: at 14:11把最近做的关于GPU上SNN稀疏加速的东西做一下总结，虽然不太成功。
T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edge
Published: at 16:23T-MAC, 用LUT加速BitNet系列的工作，在CPU上跑，后续还有一个工作叫T-MAN是在移动端的高通CPU里面的NPU上跑LUT加速。
HYTE: Flexible Tiling for Sparse Accelerators via Hybrid Static-Dynamic Approaches
Published: at 16:27ISCA2025，做稀疏数据流分块的，后半截没什么精力看了，现在的工作还没做稀疏编码。
SNN on GPU
Published: at 11:48接下来要开始着手做这个SNN在GPU上的推理加速了，写一些笔记整理思路。
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Published: at 17:47看看Shift-Window Attention。
SpikeVideoFormer: An Efficient Spike-Driven Video Transformer with Hamming Attention and O(T) Complexity
Published: at 16:56用汉明距离替换Attention中的点乘操作，避免出现Spike错开的情况。中间的做法比较有趣，但是实验感觉做的一般般，尤其是claim了自己有硬件实现的情况下energy计算还用的是纯算法的计算，并且FPGA的具体实现也没有透露，说了也没有说清楚。精度没有超过ANN2SNN的SOTA。重点还是需要用一些其他的操作替换掉对SNN不适应的算子。
Sparse Spiking Neural Network: Exploiting Heterogeneity in Timescales for Pruning Recurrent SNN
Published: at 19:11ICLR 2024 Spotlight, 利用Lyapunov Noise进行SNN Pruning。
Prosperity: Accelerating Spiking Neural Networks via Product Sparsity
Published: at 16:52HPCA在投的一篇SNN加速器文章，里面的“Product Sparsity”本质是减少相同内容的重复计算，和一般讨论的稀疏是两种不同的概念。
Towards Scalable GPU-Accelerated SNN Training via Temporal Fusion
Published: at 14:34意义不明，用Layer-By-Layer写了一下LIF就没别的Contribution了，发在了一个叫做ICANN的会上。工作量也太小了。
Recurrent Residual Module for Fast Inference in Videos
Published: at 15:25CVPR2018， DiffEncode + 稀疏加速，但感觉太老了。
Efficient Spatially Sparse Inference for Conditional GANs and Diffusion Models
Published: at 14:18NIPS2022上一篇比较有影响力的论文，对GAN和扩散模型做推理加速的工作，提出了Spatially Sparse Inference，仅在被编辑区域上稀疏地应用卷积滤波器，同时对未编辑区域复用缓存的特征
SlowFast Networks for Video Recognition
Updated: at 06:15Published: at 16:57多分支CNN，会不会有一些分支能学到更加相似的帧间变化？
DeltaCNN: End-to-End CNN Inference of Sparse Frame Differences in Videos
Updated: at 15:07Published: at 12:11利用CNN Layer的“线性”特征在帧之间做feature的差分，并且做了CUDA加速。和ViStream几乎一样的思路，能不能解决我们现在的问题？
Phi: Leveraging Pattern-based Hierarchical Sparsity for High-Efficiency Spiking Neural Networks
Published: at 17:45ISCA 2025, 基于结构化稀疏的SNN加速器。如果直接用LUT存，可能会出现需要保存的稀疏pattern数量太多，显存占用太严重，所以通过预先校准一级“结构化稀疏”，将Online Spike Activation变成一级可以完全用LUT算的L1 Sparse和稀疏度非常高的L2 Sparse。模仿一下idea搬到GPU上来做？
Temporal Flexibility in Spiking Neural Networks: Towards Generalization Across Time Steps and Deployment Friendliness
Published: at 15:38ICLR2025 Poster，似乎也在做Elastic inference？
A Simple Framework for Contrastive Learning of Visual Representations
Published: at 13:42对比学习SimCLR的论文。对比学习能对齐每一层的Feature吗？
QKFormer: Hierarchical Spiking Transformer using Q-K Attention
Published: at 18:09QKFormer，NIPS2024 Spotlight，把Direct Training SNN在ImageNet和CIFAR上的点刷的特别高，感觉之后要做就避不开它。
Transformers without Normalization
Published: at 16:09何恺明新作，用DyT代替Norm，把同步操作变成了Element Wise的操作。新文章里面有用到，学习一下。
Visualizing and Understanding the Effectiveness of BERT
Published: at 10:21最近做SNN训练的过程中在研究怎么可视化训练过程中的Loss，在想新加入的方法会不会对模型的Loss Landscape有影响，一般讲Loss Landscape怎么做可视化的文章都会引用这篇文章对Loss Landscape的分析和做法。
One-Minute Video Generation with Test-Time Training
Published: at 18:17最近Demo很火的TTT视频生成，可以生成60s级别的长视频。学习一下TTT的东西，SNN的On-Chip Learning和TTT能不能做结合？
Evolution Strategies as a Scalable Alternative to Reinforcement Learning
Published: at 16:40这两天在弄SNN训练的事情，需要验证一下用的Surrogate Gradient的准确性，老师介绍读一下这篇文章，用Evolution Strategy验证一下现在梯度估计的准确性。
SparTA: Deep-Learning Model Sparsity via Tensor-with-Sparsity-Attribute
Published: at 11:06sparTA，带稀疏优化的DNN编译器，把tensor的稀疏性作为一种重要属性考虑到编译过程中，生成高效的代码。
Scalable Diffusion Models with Transformers
Published: at 16:29Diffusion Transformer.
初探AI Infra
Updated: at 18:30Published: at 16:04趁最近找实习的机会学习、总结一下之前零散接触过的模型推理/训练加速的知识，还有一些CUDA编程的体系架构之类的内容。
Conv2Former: A Simple Transformer-Style ConvNet for Visual Recognition
Updated: at 14:57Published: at 14:39使用大kernel DS卷积替代self-attention。字节新加坡的工作。
SpikeCV: Open a Continuous Computer Vision Era
Updated: at 14:57Published: at 15:33事件相机开源框架。
Neuromorphic computing at scale
Updated: at 14:57Published: at 22:11发在Nature上的一篇review，讨论了SNN/神经模态计算社区现在面临的一些问题、挑战，和一些可能的发展方向。
Titans: Learning to Memorize at Test Time
Updated: at 14:57Published: at 18:36从TTT改进而来的新架构，尝试通过TTT的方式改进模型的记忆能力。
Segment Anything
Updated: at 14:57Published: at 13:48Meta的SAM。
SDiT: Spiking Diffusion Model with Transformer
Updated: at 14:57Published: at 14:10脉冲Diffusion Transformer，里面的Transformer的结构是RWKV的。
2024
Updated: at 14:57Published: at 12:452024.
ConvUNeXt:An efficient convolution neural network for medical image segmentation
Updated: at 14:57Published: at 15:59ConvNext + UNet，发在一个C刊上，借鉴学习一下，想想我的模块怎么设计。
Rethinking the Membrane Dynamics and Optimization Objectives of Spiking Neural Networks
Updated: at 14:57Published: at 15:23NIPS2024。主要研究的是静态任务中，推理前膜电位初始值设置对精度的影响。
ConvNext V2: Co-designing and Scaling ConvNets with Masked Autoencoders
Updated: at 14:57Published: at 06:05ConvNext续作，引入了MAE。
A ConvNet for the 2020s
Updated: at 14:57Published: at 15:22CVPR2022。Meta的工作，在ViT相关工作占视觉大头的情况下重构纯卷积的网络，并且取得了很好的效果。
LoCC工作总结
Updated: at 15:05Published: at 02:21老板找到idea到交稿只用了两个星期，第一次完整跟着做完一整篇论文的工作。
Were RNNs All We Needed?
Updated: at 15:06Published: at 16:07改进RNN，便于scale up
SparseRT: Accelerating Unstructured Sparsity on GPUs for Deep Learning Inference
Updated: at 15:06Published: at 14:18GPU上做MM相关的算子生成，利用load balancing和稀疏做加速，根据model生成PTX代码
VPRTempo: A Fast Temporally Encoded Spiking Neural Network for Visual Place Recognition
Updated: at 15:06Published: at 15:34ICRA2024的论文，用Temporal Encoding的STDP Direct Training的SNN做场景识别的任务。太简单了
Memory-Efficient Reversible Spiking Neural Networks
Updated: at 15:06Published: at 14:35通过设计提高训练速度，降低显存占用的工作。
SpikeMba: Multi-Modal Spiking Saliency Mamba for Temporal Video Grounding
Updated: at 15:06Published: at 16:40SNN+Mamba完成TVG时序视频定位任务，哈工大和北大的工作。
Integer-Valued Training and Spike-Driven Inference Spiking Neural Network for High-performance and Energy-efficient Object Detection
Updated: at 15:06Published: at 12:46SpikeYOLO，中科院自动化所的工作，ECCV2024 Oral
SNN视频流任务调研
Updated: at 10:14Published: at 13:42学习一下视频stream上任务的一些工作，大概计划一下后续的工作。
SpikeZIP-TF: Conversion is All You Need for Transformer-based SNN
Updated: at 15:06Published: at 14:19游康师兄的工作，ANN2SNN的Transformer。
SpikingJelly: An open-source machine learning infrastructure platform for spike-based intelligence
Updated: at 15:06Published: at 19:49北大惊蛰，非常有影响力的SNN框架，实现了从数据编码、数据集整合到训练、硬件部署的全流程，SNN的torch级别的工作。发表在Science Advanced上。
I-LLM: Efficient Integer-Only Inference for Fully-Quantized Low-Bit Large Language Models
Updated: at 15:06Published: at 17:11LLM的Interger-Only PTQ量化工作。
程序语言理论笔记
Updated: at 15:06Published: at 15:12程序语言理论课程的复习笔记。
The Minimum Equivalent DNF Problem and Shortest Implicants
Updated: at 15:06Published: at 06:55证明MIN-DNF问题是完全的
I-ViT: Integer-only Quantization for Efficient Vision Transformer Inference
Updated: at 15:06Published: at 15:56对ViT的纯整型量化，W8A8，中科院2023 ICCV
Efficient and Effective Methods for Mixed Precision Neural Network Quantization for Faster, Energy-efficient Inference
Updated: at 15:06Published: at 16:28EAGL，声称只要用CPU在3秒内就能完成对ResNet的量化，效率远高于HAWQ等其他传统的方法
Towards spike-based machine intelligence with neuromorphic computing
Updated: at 15:06Published: at 18:43Nature上关于SNN的综述
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
Updated: at 15:06Published: at 13:27Flash Attention，利用硬件结构加速Attention计算速度、减少内存占用的算法。核心是Tiling，Online Softmax和Kernel Fusion。
WWW: What, When, Where to Compute-in-Memory
Updated: at 15:06Published: at 18:34一些关于存内计算的验证与思考。
Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference
Updated: at 15:06Published: at 18:33谷歌的，第一篇完整跑通interger-only量化推理流程的工作。
SpikeSim: An end-to-end Compute-in-Memory Hardware Evaluation Tool for Benchmarking Spiking Neural Networks
Updated: at 15:06Published: at 18:33SNN部署的硬件设计or evaluation benchmark。
PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU
Updated: at 15:06Published: at 18:32From IPADS, 利用模型预测LLM中需要激活的MoE or Neuron，减少资源消耗。
Evaluating Spatial Accelerator Architectures with Tiled Matrix-Matrix Multiplication
Updated: at 15:06Published: at 18:31GEMM data mapping的介绍，主要是各种脉动阵列相关的加速器。
HAWQ: Hessian Aware Quantization of Neural Networks with Mixed-Precision
Updated: at 15:06Published: at 18:30模型量化经典方法，基于黑森矩阵，一种二阶信息的量化方法。
Optimizing Bit-Serial Matrix Multiplication for Reconfigurable Computing
Updated: at 15:06Published: at 18:29BISMO优化。
TVM: An Automated End-to-End Optimizing Compiler for Deep Learning
Updated: at 15:06Published: at 18:29TVM。
Roofline: An Insightful Visual Performance Model for Floating-Point Programs and Multicore Architectures
Updated: at 15:06Published: at 17:49Roofline model，描述一个系统的性能是受内存制约还是受计算制约。
A Comprehensive Survey on Electronic Design Automation and Graph Neural Networks: Theory and Applications
Updated: at 15:06Published: at 17:42图神经网络在EDA领域应用的综述。
A Hardware-Software Blueprint for Flexible Deep Learning Specialization
Updated: at 15:06Published: at 16:33VTA。
BISMO: A Scalable Bit Serial Matrix Multiplication Overlay for Reconfigurable Computing
Updated: at 15:06Published: at 14:31BISMO。
Code Transpilation for Hardware Accelerators
Updated: at 15:06Published: at 14:29基于Metalift，做的还很不完善。