SpeechLM：语音与未配对文本双分支联合预训练的技术解析与 ASR/ST 实战指南

SpeechLM：语音与未配对文本双分支联合预训练的技术解析与 ASR/ST 实战指南

发布时间：2026/9/14 22:34:35
SpeechLM：语音与未配对文本双分支联合预训练的技术解析与 ASR/ST 实战指南 SpeechLM语音与未配对文本双分支联合预训练的技术解析与 ASR/ST 实战指南【免费下载链接】unilmLarge-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities项目地址: https://gitcode.com/GitHub_Trending/un/unilm本文围绕 SpeechLMEnhanced Speech Pre-Training with Unpaired Textual Data的官方使用文档展开讲解如何利用其清洗后的 checkpoint 直接提取各层语音表征并完整覆盖 ASRLibriSpeech、语音翻译CoVoST-2微调与从零预训练的全套命令与配置同时结合仓库源码深入剖析“语音分支 单元编码器文本分支”双分支架构、预训练损失构成与 Base/Large 配置差异帮助读者既能照抄命令跑通流程也能理解每个参数在底层代码中的实际作用。1. SpeechLM 的核心思路SpeechLM 针对自监督语音预训练长期依赖“大规模带转录音频”的瓶颈提出了一个关键机制在预训练阶段同时利用未配对unpaired文本数据。其做法是将文本先转换tokenize成与语音同构的“离散单元序列”音素序列 phn 或隐藏单元序列 km再喂给一个与语音编码器共享目标的单元编码器unit encoder使模型在只有语音、只有文本、以及两者混合的 batch 上都能进行掩码预测训练。仓库中这一机制由以下文件共同实现speechlm/speechlm/models/speechlm.py完整训练版模型joint_sc2t_pretrainingtask 使用speechlm/speechlm/tasks/joint_sc2t_pretrain.py联合“语音 文本”数据的任务定义speechlm/speechlm/criterions/speechlm_criterion.py多目标损失函数speechlm/SpeechLM.py面向推理的独立精简版合并了全部特征提取所需代码不依赖 fairseq。文档中的模型命名规则为SpeechLM-Pphoneme 音素单元与SpeechLM-Hhidden-unit 隐藏单元Base960h LibriSpeech 预训练与Large60k h LibriLight 预训练。所有模型均额外混合了 40M 规模的文本数据进行预训练。2. 预训练与微调模型清单原始文档给出的完整模型列表如下下载链接分别托管于 Google Drive 与 OneDrive具体地址以 speechlm/speechlm_READme.md 中的链接为准模型预训练数据微调数据SpeechLM-P Base960h LibriSpeech 40M 文本无原始预训练模型SpeechLM-P Base960h LibriSpeech 40M 文本100h LibriSpeechASRSpeechLM-H Base960h LibriSpeech 40M 文本无原始预训练模型SpeechLM-H Base960h LibriSpeech 40M 文本100h LibriSpeechASRSpeechLM-P Base960h LibriSpeech 40M 文本CoVoST-2 En→DeSpeechLM-P Base960h LibriSpeech 40M 文本CoVoST-2 En→CaSpeechLM-P Base960h LibriSpeech 40M 文本CoVoST-2 En→ArSpeechLM-P Base960h LibriSpeech 40M 文本CoVoST-2 En→TrSpeechLM-P Large60k h LibriLight 40M 文本无原始预训练模型SpeechLM-P Large60k h LibriLight 40M 文本960h LibriSpeechASRSpeechLM-P Large60k h LibriLight 40M 文本CoVoST-2 En→De / En→Ca / En→Ar / En→Tr此外文档还单独提供了清洗版cleanedcheckpoint移除非推理必需模块后重新打包共三个SpeechLM-P Base、SpeechLM-H Base、SpeechLM-P Large。这些 checkpoint 是下一节特征提取的入口因为 speechlm/SpeechLM.py 就是按它们的状态字典结构来组织的。3. 使用清洗版 Checkpoint 提取语音特征文档给出的官方用法可直接复制运行import torch import torch.nn.functional as F from SpeechLM import SpeechLMConfig, SpeechLM checkpoint torch.load(path/to/the/cleaned/checkpoint.pt) cfg SpeechLMConfig(checkpoint[cfg][model]) model SpeechLM(cfg) model.load_state_dict(checkpoint[model]) model.eval() wav_input_16khz torch.randn(1,10000) normalize checkpoint[cfg][task][normalize] # False for base model, True for large model if normalize: wav_input_16khz F.layer_norm(wav_input_16khz[0], wav_input_16khz[0].shape).unsqueeze(0) # extract the representation of last layer rep model.extract_features(wav_input_16khz)[0] # extract the representation of each layer output_layer model.cfg.encoder_layers model.cfg.text_transformer.encoder.layers rep, layer_results model.extract_features(wav_input_16khz, output_layeroutput_layer, ret_layer_resultsTrue)[0] layer_reps [x.transpose(0, 1) for x in layer_results]使用要点输入是 16 kHz 原始波形形状(B, T)无需提特征模型内部由卷积特征提取器完成下采样。normalize 开关必须与预训练时一致从预训练配置看Base 模型 speechlm/speechlm/config/pretrain/speechlm_base_librispeech.yaml 中task.normalize: falseLarge 模型 speechlm/speechlm/config/pretrain/speechlm_large_librilight.yaml 中为normalize: trueextractor_mode也从default换成了layer_norm。因此 Large 模型推理前必须做逐样本 layer_normBase 模型不能做。逐层表征提取output_layer encoder_layers text_transformer.encoder.layers表示取“最后一层”。这里暴露了架构信息——SpeechLM 的完整前向包含语音编码器 单元编码器两段串联所以层编号是两段相加。extract_features(..., ret_layer_resultsTrue)返回所有中间层layer_results中每个张量形状为(T, B, D)示例代码将其转置回(B, T, D)方便使用。3.1 源码解读SpeechLM.py的推理路径SpeechLMConfig 定义了全部结构超参的默认值被 checkpoint 中的cfg[model]覆盖。几个值得注意的参数label_rate默认 5020 ms 帧率但 Base 预训练脚本实际以model.label_rate10010 ms覆盖见 base_speechlmp.shconv_feature_layers: [(512,10,5)] [(512,3,2)] * 4 [(512,2,2)] * 2wav2vec 2.0 风格的 8 层卷积总下采样率为 5×2^6320即 16 kHz 波形约每 20 ms 一个特征帧final_dim: 256预测投影维度add_unit_encoder / mix_with_unit / use_pred_unit / l2_embedding控制文本分支行为。SpeechLM.forward按输入分派两条路径speechlm/SpeechLM.py#L419-L446forward_speech卷积特征提取 → 掩码 → 语音 Transformer 编码器 → 若add_unit_encoderTrue把语音编码器输出经convert_embeddings与单元嵌入混合后送入unit_encoder第二段 Transformer 编码器并对两段输出分别计算掩码预测 logitsforward_text纯文本路径将单元 token 嵌入后走同样的掩码预测用于训练时的纯文本 batch。特征提取走extract_featuresspeechlm/SpeechLM.py#L590-L633以features_onlyTrue, maskFalse前向语音输出为x当output_layer超过cfg.encoder_layers时会自动把语音编码器输出接入unit_encoder继续前向这正是能取到“第 6~18 层语音 6 层 文本 12 层”这类层索引的原因。另外forward_targets中的feat2tar_ratio label_rate × 320 / 16000speechlm/SpeechLM.py#L153-L155用于把特征帧与标签序列对齐推理时不需要标签故可忽略。4. 环境准备Setup文档给出的环境搭建步骤git submodule update --init SpeechLM/fairseq cd SpeechLM/ pip install --editable fairseq/ pip install sacrebleu1.5.1对应当前仓库的实际情况需要说明两点本仓库中 SpeechLM 位于speechlm/目录下fairseq 是 git submodule仓库快照中 speechlm/fairseq 为空目录——必须先执行上述git submodule update --init拉取依赖才能训练/微调文档中的命令路径SpeechLM/...对应到本仓库即speechlm/speechlm/...例如文档写speechlm/scripts/tune_speechlm_asr/finetune_base_ctc.sh实际文件是 speechlm/speechlm/scripts/tune_speechlm_asr/finetune_base_ctc.sh。后文所有命令均按仓库实际路径给出。sacrebleu1.5.1是 ST 任务评估 BLEU 所需。训练/微调本身通过 fairseq 的 Hydra 入口fairseq_cli/hydra_train.py启动配置目录指向speechlm/speechlm/config/。5. LibriSpeech 上的 ASR 微调5.1 数据准备按 wav2vec 2.0 的 manifest 规范准备train.tsv音频清单与train.ltr字母标签并确认词表与预训练模型一致。词表文件即 speechlm/dataset/LibriSpeech/asr/dict.ltr.txt示例数据见 speechlm/dataset/LibriSpeech/asr 下的train_sample100.tsv/train_sample100.ltr各取 100 条样例。把准备好的数据放入$data_dir。5.2 微调 CTC 模型Base 模型# Usage: speechlm/scripts/tune_speechlm_asr/finetune_base_ctc.sh model_path data_dir cpt_tag [mount$PWD] [world_size8] [update_freq1] model_pathpath/to/your/pre-trained/model data_dirdataset/LibriSpeech/asr bash speechlm/speechlm/scripts/tune_speechlm_asr/finetune_base_ctc.sh $model_path $data_dir tag400kLarge 模型# Usage: speechlm/scripts/tune_speechlm_asr/finetune_large_ctc.sh model_path data_dir cpt_tag [mount$PWD] [world_size8] [update_freq4] model_pathpath/to/your/pre-trained/model data_dirdataset/LibriSpeech/asr bash speechlm/speechlm/scripts/tune_speechlm_asr/finetune_large_ctc.sh $model_path $data_dir tag400k从配置文件可以读出两套微调的默认设置Base 与 Large 差异明显配置项Base100hLarge960h配置文件speechlm_base_100h.yamlspeechlm_large_960h.yaml训练/验证子集train_100/dev_othertrain_960/dev_other最大更新步数30000200000学习率1e-5sentence_avg1e-5sentence_avg归一化task.normalizefalsetrue模型 / 损失speechlm_ctcctczero_infinity: true同左学习率调度tri_stagephase_ratio [0.1, 0.4, 0.5]final_lr_scale 0.05同左掩码微调期apply_mask: truemask_prob 0.65mask_channel_prob 0.5长度 64feature_grad_mult: 0.0冻结特征提取器同左其中model.w2v_path指向预训练权重由脚本传入best_checkpoint_metric: wer表示按 WER 保留最佳 checkpoint。feature_grad_mult: 0.0意味着微调时卷积特征提取器不更新这与 SpeechLM.py 中forward_features的分支一致系数为 0 时在torch.no_grad()下提特征。5.3 解码直接 CTC 解码Viterbi# Usage: speechlm/scripts/tune_speechlm_asr/inference_ctc.sh model_path data_dir [gen-setdev_clean,dev_other,test_clean,test_other] model_pathpath/to/your/fine-tuned/model data_dirdataset/LibriSpeech/asr bash speechlm/speechlm/scripts/tune_speechlm_asr/inference_ctc.sh $model_path $data_dir # for large models # bash speechlm/speechlm/scripts/tune_speechlm_asr/inference_ctc_large.sh $model_path $data_dir4-gram 语言模型解码基于 Flashlight 与 kenlm 实现。需把 LibriSpeech LM 语料的 4-gram arpa 模型与 word-to-letter 词典librispeech_lexicon.lst文档中给出托管链接放入$data_dir然后# Usage: speechlm/scripts/tune_speechlm_asr/inference_ctc_kenlm.sh model_path data_dir [gen-setdev_clean,dev_other,test_clean,test_other] bash speechlm/speechlm/scripts/tune_speechlm_asr/inference_ctc_kenlm.sh $model_path $data_dirLarge 模型 fairseq 词级 LM 解码把lm_librispeech_word_transformer.pt及其词表dict.txt放入$data_dir/fairseq_word_lmdict.txt需大写化处理以兼容 word-to-letter 词典word-to-letter 词典放入$data_dir# Usage: speechlm/scripts/tune_speechlm_asr/inference_ctc_large_fsqlm.sh model_path data_dir [gen-setdev_clean,dev_other,test_clean,test_other] bash speechlm/speechlm/scripts/tune_speechlm_asr/inference_ctc_large_fsqlm.sh $model_path $data_dir dev_other解码侧的三种解码器配置分别对应 speechlm/speechlm/config/decode/ 下的infer_viterbi.yaml、infer_kenlm.yaml、infer_fsqlm.yaml。6. CoVoST-2 上的语音翻译ST微调6.1 数据准备下载 Common Voice 4 版英语音频到$cv_root/en运行清单生成脚本完成 mp3→波形转换、生成语音-翻译 tsv、生成数据配置文件langde # ca,ar,tr cv_rootdataset/CommonVoice/v4 bash speechlm/speechlm/data_process/prepare_covost2_enxx.sh $lang $cv_root生成逻辑见 speechlm/speechlm/data_process/prepare_covost2_enxx.sh 与 covost2/prepare_covost_data.py、covost2/mp3_to_wav.py。文档提供的示例数据在 speechlm/dataset/CommonVoice/v4/en/en-de其中config_base_ende.yaml 指定 sentencepiece 模型spm_char_st_en_de.model、字符级词表spm_char_st_en_de.txt、16 kHz 采样、use_audio_input: truedev-sample100_st_en_de_local.tsv是 100 条的 dev 样例清单。6.2 微调 Encoder-Decoder 模型Base 模型产出存于$mount/exp/finetune_covostmodel_pathpath/to/your/pre-trained/model langde # ca,ar,tr data_dirdataset/CommonVoice/v4/en/en-${lang} # Usage (Base model): speechlm/scripts/tune_speechlm_st/ft_base_covost_enxx.sh model_path data_dir lang cpt-tag [mount$PWD] [world_size8] [update_freq2] bash speechlm/speechlm/scripts/tune_speechlm_st/ft_base_covost_enxx.sh $model_path $data_dir $lang tag400kLarge 模型默认update_freq4# Usage (Large model): speechlm/scripts/tune_speechlm_st/ft_large_covost_enxx.sh model_path data_dir lang cpt-tag [mount$PWD] [world_size8] [update_freq4] bash speechlm/speechlm/scripts/tune_speechlm_st/ft_large_covost_enxx.sh $model_path $data_dir $lang tag400k从源码结构看ST 模型是 speechlm/speechlm/models/speechlm_st.py 中的编码器-解码器结构其配置类SpeechLMS2TConfig继承自 Hubert 系 ASR 配置新增decoder_embed_dim: 768、decoder_layers: 6、decoder_attention_heads: 12等参数并默认开启解码器相对位置编码use_rel_pos_enc: true即复用预训练的语音编码器再接一个自回归解码器做翻译。6.3 解码# Usage: speechlm/scripts/tune_speechlm_st/inference_base.sh model_path data_dir lang [gen-setdev] [beam_size5] model_pathpath/to/your/fine-tuned/model langde # ca,ar,tr data_dirdataset/CommonVoice/v4/en/en-${lang} bash speechlm/speechlm/scripts/tune_speechlm_st/inference_base.sh $model_path $data_dir $lang devLarge 模型同理# Usage: speechlm/scripts/tune_speechlm_st/inference_large.sh model_path data_dir lang [gen-setdev] [beam_size5] bash speechlm/speechlm/scripts/tune_speechlm_st/inference_large.sh $model_path $data_dir $lang dev默认 beam_size5评估 BLEU 依赖第 4 节安装的 sacrebleu。7. 从零预训练7.1 三类模型的预训练脚本所有预训练均要求先按下一节“Tokenizers”准备语音单元标签与文本单元数据。产出 checkpoint 存于$mount/pretrain。SpeechLM-P Base脚本 base_speechlmp.shdata_dirdataset/LibriSpeech/phone_unit # should contain train_960.{tsv,phn} text_data_dirdataset/LibriLM/phone_unit/bin-idx # should contain train_text.phn-ltr.{phn,ltr}.{bin,idx} # Usage: speechlm/scripts/pretrain_speechlm/base_speechlmp.sh data_dir text_data_dir [mount$PWD] [world_size32] [update_freq1] bash speechlm/speechlm/scripts/pretrain_speechlm/base_speechlmp.sh $data_dir $text_data_dirSpeechLM-H Basedata_dirdataset/LibriSpeech/hidden_unit # should contain train_960.{tsv,phn} text_data_dirdataset/LibriLM/km-ltr/bin-idx # should contain train_text.km-ltr.{km,ltr}.{bin,idx} # Usage: speechlm/scripts/pretrain_speechlm/base_speechlmh.sh data_dir text_data_dir [mount$PWD] [world_size32] [update_freq1] bash speechlm/speechlm/scripts/pretrain_speechlm/base_speechlmp.sh $data_dir $text_data_dir注意原文档此处 Usage 注释写的是base_speechlmh.sh但命令体复制的是base_speechlmp.sh从仓库中确实存在独立的 base_speechlmh.sh 脚本来推断命令行应为base_speechlmh.sh原文此处疑似笔误。SpeechLM-P Large脚本 large_speechlmp.shdata_dirdataset/LibriSpeech/phone_unit # should contain train_960.{tsv,phn} text_data_dirdataset/LibriLM/phone_unit/bin-idx # should contain train_text.phn-ltr.{phn,ltr}.{bin,idx} # Usage: speechlm/scripts/pretrain_speechlm/large_speechlmp.sh data_dir text_data_dir [mount$PWD] [world_size32] [update_freq1] bash speechlm/speechlm/scripts/pretrain_speechlm/large_speechlmp.sh $data_dir $text_data_dir以 Base 脚本为例其内部调用为base_speechlmp.sh#L19-L40python $CODE_ROOT/fairseq/fairseq_cli/hydra_train.py \ --config-dir $CODE_ROOT/speechlm/config/pretrain \ --config-name speechlm_base_librispeech \ common.user_dir$CODE_ROOT/speechlm \ task.labels[phn] \ model.label_rate100 \ task.data$DATA_DIR task.label_dir$DATA_DIR \ task.text_cfg.text_data$TEXT_DATA_DIR \ dataset.train_subsettrain_960train_text.phn-ltr \ dataset.valid_subsetdev_cleandev_clean.phn-ltr \ dataset.max_tokens1400000 \ distributed_training.distributed_world_size${world_size} \ optimization.update_freq[${update_freq}] \ ...几个值得注意的实现细节dataset.train_subsettrain_960train_text.phn-ltr号把语音子集与文本子集拼成一个联合 batch 流这正是“语音与未配对文本联合训练”在数据层的落点对应 task 实现 joint_sc2t_pretrain.py 与数据包装器 speechlm/speechlm/data/multimodal_corpus_dataset.py、concat_dataset.pytask.labels[phn]与model.label_rate100P 模型以音素为标签、10 ms 帧率脚本还会校验当前目录必须位于SpeechLM代码根下模型输出目录为${mount}/exp/pretrain/base_speechlmp_${world_size}gpu_${update_freq}accum。7.2 Base 与 Large 训练配置对照两份预训练配置的差异base vs large配置项BaseLibriSpeechLargeLibriLighttask.normalize/extractor_modefalse / defaulttrue / layer_norm语音编码器6 层 × 768 维 × 8 头FFN 307212 层 × 1024 维 × 16 头FFN 4096文本单元编码器6 层 × 768 维 × 8 头12 层 × 1024 维 × 16 头mask_prob0.800.80feature_grad_mult0.1特征提取器降速训练1.0dropout 系列0.10.0改用layer_norm_first: truemax_tokens1400000900000初始学习率 / clip_norm5e-4 / 10.01e-3 / 1.0scaling_for_att默认 1.032防大模型注意力溢出其他共用max_update: 400000Adam β(0.9,0.98) ε1e-06weight_decay 0.01polynomial_decay 调度、warmup 32000max_sample_size: 250000/min_sample_size: 3200016 kHz 随机裁剪同左文本侧text_cfg配置两版一致data_config: config.yaml、tokens_per_sample: 1024、shorten_method: random_crop、text_maxtokens_ratio: 1.0即每条文本最长 1024 单元 token超长随机裁剪。7.3 预训练损失构成损失由 speechlm_criterion.py 汇总配置文件中的默认权重为pred_masked_weight: 1.0/pred_nomask_weight: 0.0只对被掩码帧计交叉熵HuBERT 式掩码预测loss_weights: [10,]第二段目标单元编码器输出 → 文本侧单元的权重即总损失为语音分支掩码损失 10 × 单元分支掩码损失text_ctc_weight: 0.1文本单元序列上的 CTC 辅助损失源码中model.add_text_ctc: true时unit_encoder_ctc_head产生encoder_out_ctcSpeechLM.py 的 forward_text 对应逻辑text_mum_weight: 0.0masked unit modeling 权重两版配置均未启用compute_mum: false。mix_with_unit: true则对应 convert_embeddings以mask_prob/2的比率把语音表征随机替换为单元嵌入迫使语音编码器输出“单元可替换”这是让语音与文本两个模态在表征空间对齐的关键机制。8. Tokenizer 体系把“未配对文本”变成语音同构单元8.1 语音侧音素 TokenizerPhoneme-unit对无标注语音产生帧对齐音素实现方式是一个混合 HMM ASR 模型Base 设定下用 100h LibriSpeech 标注数据在 Kaldi recipe 下训练 HMM再解码未配对语音从 lattice 中取出对齐音素。文档直接提供了 960h 处理好的音素结果train_960.tsv/.phn、dev_clean.tsv/.phn托管链接见 README标签帧率为 10010 ms。仓库内对应样例speechlm/dataset/LibriSpeech/phone_unit/train_sample100.phn 与 dict.phn.txt。音素生成的辅助工具链位于 speechlm/speechlm/data_process/phoneme_tokenizer/ltr2kaldi_phn_sil025.py、repeat_withou_insert_sil_less_4375.py等负责字母→音素转换、静音切分等预处理。8.2 文本侧音素 Tokenizer把未配对文本转为 (phonemes, letters) 配对数据流水线为words - phonemes - upsampled phones。运行一键脚本会下载 LibriSpeech LM 语料并产出train_text.phn-ltr.phn.{idx,bin}与train_text.phn-ltr.ltr.{idx,bin}# data will be in dataset/LibriLM/phone_unit/ bash speechlm/speechlm/data_process/prepare_phn2ltr_librilm.sh运行前需把文档提供的音素/字母词表放入dataset/LibriLM/phone_unit/bin-idx/仓库中该目录实际文件为 dict.phn.txt 与 dict.ltr.txtREADME 原文写作dcit.phn.txt系拼写笔误另有该子集的 config.yaml。8.3 语音侧隐藏单元 TokenizerHidden-unitSpeechLM-H 变体使用离散隐藏单元替代音素。准备三部分参考 wav2vec 2.0 manifest 规范音频清单train.tsv、对齐的隐藏单元train.km、单元词表dict.km.txt。样例见 speechlm/dataset/LibriSpeech/hidden_unit/train_sample100.tsv/train_sample100.km/ dict.km.txt文本侧词表在 speechlm/dataset/LibriLM/hidden_unit/bin-idx/。8.4 文本侧隐藏单元 TokenizerFastSpeech 式用一个FastSpeech 风格的序列到序列模型原版生成连续频谱这里改为生成离散单元把未配对文本直接转换成语音式隐藏单元。训练语料仅 100h LibriSpeech把 ASR 转录转成音素序列用语音侧隐藏单元 Tokenizer 从语音中提取隐藏单元在配对数据上训练模型speechlm/speechlm/models/fasttext2unit.pydata_dirdataset/LibriSpeech/fast_phone2unit bash speechlm/speechlm/scripts/tokenizer_fastT2U/train_s_5e-4.sh $data_dir对大规模文本语料生成隐藏单元gen_setdataset/LibriSpeech/fast_phone2unit/genset_examples bash speechlm/speechlm/scripts/tokenizer_fastT2U/generate.sh $model_path $gen_set推理入口为 speechlm/speechlm/generate_unit.py / unit_generator.py。训练与生成的样例数据train_exmples.tsv、genset_examples.tsv、config.yaml、config_generate.yaml位于 speechlm/dataset/LibriSpeech/fast_phone2unittokenizer 模型 checkpoint 的下载链接见 README 原文。9. 引用信息与代码组织小结如果本研究对你的工作有帮助请引用论文SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual DataZhang, Chen, Zhou, et al., 2022article{zhang2022speechlm, title {SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Data}, author {Zhang, Ziqiang and Chen, Sanyuan and Zhou, Long and Wu, Yu and Ren, Shuo and Liu, Shujie and Yao, Zhuoyuan and Gong, Xun and Dai, Lirong and Li, Jinyu and Wei, Furu}, eprint{2209.15329}, archivePrefix{arXiv}, primaryClass{cs.CL}, year{2022} }最后给出与本文各章节对应的仓库文件地图便于按图索骥主题关键路径推理/特征提取独立实现speechlm/SpeechLM.py、speechlm/modules.py预训练模型/损失/任务speechlm/speechlm/models/speechlm.py、speechlm/speechlm/criterions/speechlm_criterion.py、speechlm/speechlm/tasks/joint_sc2t_pretrain.pyASR/ST 微调模型speechlm/speechlm/models/speechlm_ctcasr.py、speechlm/speechlm/models/speechlm_st.py训练/微调/解码配置speechlm/speechlm/config/pretrain、finetune、decode 三个子目录全部可执行脚本speechlm/speechlm/scripts/pretrain_speechlm、tune_speechlm_asr、tune_speechlm_st、tokenizer_fastT2U数据样例speechlm/dataset/LibriSpeech、LibriLM、CommonVoice适用前提提醒本指南中的命令默认在speechlm/speechlm/代码根目录下执行且 fairseq submodule 已初始化Base 与 Large 模型的normalize/归一化约定不同特征提取与微调时必须与各自预训练配置保持一致否则表征与预训练分布不符。【免费下载链接】unilmLarge-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities项目地址: https://gitcode.com/GitHub_Trending/un/unilm创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考