Note on CUDA programming

CUDA execution model

• Programming Model
• Compilation Workflow

Global memory

• Memory architecture
• Memory management
• Memory access pattern

Shared memory and constant memory

• Configurations of Shared Memory
• Bank conflict
• Synchronization of Shared Memory
• Warp Shuffles

Texture memory

• Texture Memory

Streams and concurrency

• Stream and Concurrency
• Triple Buffering

Worked Examples

• Parallel Reduction
• Matrix Addition
• Matrix Transpose
• Matrix Multiplcation

Tensor core

• Tensor core

Profiling

• Calculation of Warp Occupancy
• Performance tuning
• Nsight compute
• Nsight system

Debugging

• cuda-gdb

Mannual

nvcc CUDA C++ Programming Guide CUDA C++ Best Practices Guide Parallel Thread Execution ISA Version 8.5 CUDA runtime API CUDA driver API Thrust CUDA-GDB Compute Sanitizer Nsight System Nsight compute Profiler CUDA binary utilities

Miscellaneous

• Nvidia Technical Blog
• CUDA FAQ
• Syntax
• Thread indexing cheatsheet
• Trouble shooting
• NVIDIA A10
• kriegalex
• HangJie720
• deeplearning
• Good articles