|
|
@@ -31,6 +31,45 @@ https://huggingface.co/llm-agents
|
|
|
|
|
|
## 源码分析
|
|
|
|
|
|
-```
|
|
|
+requirements.txt
|
|
|
+
|
|
|
+flask + gradio 构建接口和用户界面。
|
|
|
+pebble 多线程
|
|
|
+accelerate PyTorch
|
|
|
+transformers
|
|
|
+
|
|
|
+### 训练数据
|
|
|
|
|
|
+设置GPU核心数，哪个模型：7B/34b，模型位置
|
|
|
```
|
|
|
+accelerate launch \
|
|
|
+ --main_process_port 18200 \
|
|
|
+ --mixed_precision bf16 \
|
|
|
+ --num_machines 1 \
|
|
|
+ --num_processes $NUM_GPUS \
|
|
|
+ --use_deepspeed \
|
|
|
+ --deepspeed_config_file $DEEPSPEED \
|
|
|
+ train/finetune.py \
|
|
|
+ --model_name_or_path ${MODEL_PATH} \
|
|
|
+ --use_slow_tokenizer \
|
|
|
+ --gradient_checkpointing \
|
|
|
+ --train_file $TRAIN_FILE \
|
|
|
+ --max_seq_length 2048 \
|
|
|
+ --preprocessing_num_workers 16 \
|
|
|
+ --per_device_train_batch_size $BATCH_SIZE_PER_GPU \
|
|
|
+ --gradient_accumulation_steps $GRADIENT_ACC_STEPS \
|
|
|
+ --learning_rate $LEARNING_RATE \
|
|
|
+ --lr_scheduler_type linear \
|
|
|
+ --warmup_ratio 0.03 \
|
|
|
+ --weight_decay 0. \
|
|
|
+ --num_train_epochs $NUM_TRAIN_EPOCHS \
|
|
|
+ --output_dir $OUTPUT_DIR \
|
|
|
+ --with_tracking \
|
|
|
+ --report_to tensorboard \
|
|
|
+ --logging_steps 1 \
|
|
|
+ --use_flash_attn \
|
|
|
+ --mask_prompt \
|
|
|
+ | tee $OUTPUT_DIR/logs.txt
|
|
|
+```
|
|
|
+
|
|
|
+执行 microsoft/ToRA/src/train/finetune.py main 函数开始训练。
|
