link管理

链接快照平台

输入网页链接，自动生成快照
标签化管理网页链接

bert4torch

v0.4.9.post2

an elegant bert4torch For more information about how to use this package README

Latest version published 29 days ago

License: MIT

PyPI

GitHub

Package Health Score

64 / 100

security
Security review needed
popularity
Small
maintenance
Healthy
community
Sustainable

Popularity

Small

GitHub Stars: 1.09K
Forks
Contributors

Direct Usage Popularity

Based on project statistics from the GitHub repository for the PyPI package bert4torch, we found that it has been starred 1,095 times.

Security

Security review needed

Security and license risk for latest version

Release Date: Mar 17, 2024
Direct Vulnerabilities: 0
C

0
H

0
M

0
L
Indirect Vulnerabilities: 0
C

1
H

1
M

3
L
License Risk: 0
H

0
M

0
L

All security vulnerabilities belong to production dependencies of direct and indirect packages.

License

Security Policy

We found a way for you to contribute to the project! Looks like bert4torch is missing a security policy. security vulnerability was detected in an indirect dependency that is added to your project when the latest version of bert4torch is installed. We highly advise you to review these security issues. You can connect your project's repository to Snyk to stay up to date on security alerts and receive automatic fix pull requests.

Fix it in your project with Snyk!

Maintenance

Healthy

Commit Frequency

Open Issues
Open PR
Last Release: 29 days ago
Last Commit: 29 days ago Further analysis of the maintenance status of bert4torch based on released PyPI versions cadence, the repository activity, and other data points determined that its maintenance is Healthy. We found that bert4torch demonstrates a positive version release cadence with at least one new version released in the past 3 months. As a healthy sign for on-going project maintenance, we found that the GitHub repository had at least 1 pull request or issue interacted with by the community.

Community

Sustainable

Readme
Contributing.md
Code of Conduct
Contributors
Funding

This project has seen only 10 or less contributors.

We found a way for you to contribute to the project! Looks like bert4torch is missing a Code of Conduct.

Embed Package Health Score Badge

Keep your project healthy

Check your requirements.txt

Ensure all the packages you're using are healthy and well-maintained

Snyk Vulnerability Scanner

Get health score & security insights directly in your IDE

Package

Python Versions Compatibility: Unspecified 2 years
Latest Release: 29 days ago
Dependencies
Versions
Maintainers
Wheels

Readme

bert4torch

注意事项 ：pip包的发布慢于git上的开发版本，git clone 注意引用路径 ，注意权重是否需要转换

测试用例 ：


                      git clone https://github.com/Tongjilibo/bert4torch

，修改example中的预训练模型文件路径和数据路径即可启动脚本

自行训练 ：针对自己的数据，修改相应的数据处理代码块

开发环境 ：原使用


                      torch==1.10

版本进行开发，现已切换到


                      torch2.0

开发，如其他版本遇到不适配，欢迎反馈

2. 功能

LLM模型 : 加载chatglm、llama、 baichuan、ziya、bloom等开源大模型权重进行推理和微调

核心功能 ：加载bert、roberta、albert、xlnet、nezha、bart、RoFormer、RoFormer_V2、ELECTRA、GPT、GPT2、T5、GAU-alpha、ERNIE等预训练权重继续进行finetune、并支持在bert基础上灵活定义自己模型

丰富示例 ：包含 llm 、 pretrain 、 sentence_classfication 、 sentence_embedding 、 sequence_labeling 、 relation_extraction 、 seq2seq 、 serving 等多种解决方案

实验验证 ：已在公开数据集实验验证，使用如下 examples数据集

易用trick ：集成了常见的 trick ，即插即用

其他特性 ：加载transformers库模型一起使用；调用方式简洁高效；有训练进度条动态展示；配合torchinfo打印参数量；默认Logger和Tensorboard简便记录训练过程；自定义fit过程，满足高阶需求

训练过程 ：

2022-10-28 23:16:10 - Start Training
2022-10-28 23:16:10 - Epoch: 1/2
5000/5000 [==============================] - 13s 3ms/step - loss: 0.1351 - acc: 0.9601
Evaluate: 100%|██████████████████████████████████████████████████| 2500/2500 [00:03&lt;00:00, 798.09it/s] 
test_acc: 0.98045. best_test_acc: 0.98045
2022-10-28 23:16:27 - Epoch: 2/2
5000/5000 [==============================] - 13s 3ms/step - loss: 0.0465 - acc: 0.9862
Evaluate: 100%|██████████████████████████████████████████████████| 2500/2500 [00:03&lt;00:00, 635.78it/s] 
test_acc: 0.98280. best_test_acc: 0.98280
2022-10-28 23:16:44 - Finish Training
Quick-Start
快速上手教程，教程示例，实战示例
bert4torch介绍(知乎)，bert4torch快速上手(知乎)，bert4torch又双叒叕更新啦(知乎)
4. 版本和更新历史
4.1 版本历史
bert4torch
torch4keras
0.4.9.post2
0.2.1.post2
增加get_weight_decay_optim_groups函数, attention中允许is_causal，修改repetition_penalty的bug，把baichuan从llama中剥离，修复config_path的bug，允许num_key_value_heads参数，torch4keras-v0.2.1.post2更新特性
20240221
0.4.8
0.2.0
fastapi发布服务允许闲时offload到cpu, build_transformer_model允许从hf下载, 添加FillMask的pipeline, 添加SequenceClassificationTrainer
20240204
0.4.7
0.1.9
修改save_pretrained用于保存文件夹, 增加GenerateSpeed用于统计token生成速度，修复t5在use_states=True时候的错误, 修改层次编码的bug, 增加deepseek_moe模型，修复generation并发错误，优化大模型耗时
20240116
0.4.6
0.1.8
bug修复，增加save_pretrained用于保存transformer格式的权重, 增加部分embedding模型
4.2 更新历史
5. 预训练权重
预训练模型支持多种代码加载方式
from bert4torch.models import build_transformer_model
# 1. 仅指定config_path: 从头初始化模型结构, 不加载预训练模型
model = build_transformer_model('./model/bert4torch_config.json')
# 2. 仅指定checkpoint_path: 
## 2.1 文件夹路径: 自动寻找路径下的*.bin/*.safetensors权重文件 + bert4torch_config.json/config.json文件
model = build_transformer_model(checkpoint_path='./model')
## 2.2 文件路径/列表: 文件路径即权重路径/列表, config会从同级目录下寻找
model = build_transformer_model(checkpoint_path='./pytorch_model.bin')
## 2.3 model_name: hf上预训练权重名称, 会自动下载hf权重以及bert4torch_config.json文件
model = build_transformer_model(checkpoint_path='bert-base-chinese')
# 3. 同时指定config_path和checkpoint_path(本地路径名或model_name排列组合): 
config_path = './model/bert4torch_config.json'  # 或'bert-base-chinese'
checkpoint_path = './model/pytorch_model.bin'  # 或'bert-base-chinese'
model = build_transformer_model(config_path, checkpoint_path)
github，junnyu/wobert_chinese_base，junnyu/wobert_chinese_plus_base
wobert_chinese_base, wobert_chinese_plus_base
roberta
chinese-roberta-wwm-ext
github，hfl/chinese-roberta-wwm-ext, hfl/chinese-roberta-wwm-ext-large
chinese-roberta-wwm-ext, chinese-roberta-wwm-ext-large
roberta-small/tiny
github，Tongjilibo/chinese_roberta_L-4_H-312_A-12, Tongjilibo/chinese_roberta_L-6_H-384_A-12
albert
brightmart
github，torch, voidful/albert_chinese_tiny，voidful/albert_chinese_small, voidful/albert_chinese_base, voidful/albert_chinese_large, voidful/albert_chinese_xlarge, voidful/albert_chinese_xxlarge
albert_chinese_tiny，albert_chinese_small, albert_chinese_base, albert_chinese_large, albert_chinese_xlarge, albert_chinese_xxlarge
nezha
NEZHA
github，torch, sijunhe/nezha-cn-base, sijunhe/nezha-cn-large, sijunhe/nezha-base-wwm, sijunhe/nezha-large-wwm
nezha-cn-base, nezha-cn-large, nezha-base-wwm, nezha-large-wwm
nezha_gpt_dialog
bojone
github, Tongjilibo/nezha_gpt_dialog
xlnet
chinese-xlnet
github, hfl/chinese-xlnet-base
chinese-xlnet-base
tranformer_xl
huggingface
transfo-xl/transfo-xl-wt103
transfo-xl-wt103
deberta
Erlangshen-DeBERTa-v2
IDEA-CCNL/Erlangshen-DeBERTa-v2-97M-Chinese, IDEA-CCNL/Erlangshen-DeBERTa-v2-320M-Chinese, IDEA-CCNL/Erlangshen-DeBERTa-v2-710M-Chinese
Erlangshen-DeBERTa-v2-97M-Chinese, Erlangshen-DeBERTa-v2-320M-Chinese, Erlangshen-DeBERTa-v2-710M-Chinese
electra
Chinese-ELECTRA
github，hfl/chinese-electra-base-discriminator
chinese-electra-base-discriminator
ernie
ernie
paddle，nghuyong/ernie-1.0-base-zh, nghuyong/ernie-3.0-base-zh
ernie-1.0-base-zh, ernie-3.0-base-zh
roformer
roformer
github，junnyu/roformer_chinese_base
roformer_chinese_base
roformer_v2
github，junnyu/roformer_v2_chinese_char_base
roformer_v2_chinese_char_base
simbert
simbert
github，Tongjilibo/simbert-chinese-base, Tongjilibo/simbert-chinese-small, Tongjilibo/simbert-chinese-tiny
simbert_v2/roformer-sim
github，junnyu/roformer_chinese_sim_char_base，junnyu/roformer_chinese_sim_char_ft_base，junnyu/roformer_chinese_sim_char_small，junnyu/roformer_chinese_sim_char_ft_small
roformer_chinese_sim_char_base, roformer_chinese_sim_char_ft_base, roformer_chinese_sim_char_small, roformer_chinese_sim_char_ft_small
GAU-alpha
github, Tongjilibo/chinese_GAU-alpha-char_L-24_H-768
github, torch, Tongjilibo/uie-base
CDial-GPT
thu-coai
github, thu-coai/CDial-GPT_LCCC-base, thu-coai/CDial-GPT_LCCC-large
CDial-GPT_LCCC-base, CDial-GPT_LCCC-large
cmp_lm(26亿)
github, TsinghuaAI/CPM-Generate
CPM-Generate
nezha_gen
huawei_noah
github, Tongjilibo/chinese_nezha_gpt_L-12_H-768_A-12
chatglm-6b
THUDM
github, THUDM/chatglm-6b, THUDM/chatglm-6b-int8, THUDM/chatglm-6b-int4, v0.1.0
chatglm-6b, chatglm-6b-int8, chatglm-6b-int4, chatglm-6b-v0.1.0
chatglm2-6b
THUDM
github, THUDM/chatglm2-6b, THUDM/chatglm2-6b-int4, THUDM/chatglm2-6b-32k
chatglm2-6b, chatglm2-6b-int4, chatglm2-6b-32k
chatglm3-6b
THUDM
github, THUDM/chatglm3-6b, THUDM/chatglm3-6b-32k
chatglm3-6b, chatglm3-6b-32k
llama
llama
github
llama-7b, llama-13b
llama-2
github, meta-llama/Llama-2-7b-hf, meta-llama/Llama-2-7b-chat-hf, meta-llama/Llama-2-13b-hf, meta-llama/Llama-2-13b-chat-hf
Llama-2-7b-hf, Llama-2-7b-chat-hf, Llama-2-13b-hf, Llama-2-13b-chat-hf
chinese_llama_alpaca
github
chinese_alpaca_plus_7b, chinese_llama_plus_7b
Belle_llama
LianjiaTech
github, BelleGroup/BELLE-LLaMA-7B-2M-enc
合成说明、BELLE-LLaMA-7B-2M-enc
IDEA-CCNL
IDEA-CCNL/Ziya-LLaMA-13B-v1, IDEA-CCNL/Ziya-LLaMA-13B-v1.1, IDEA-CCNL/Ziya-LLaMA-13B-Pretrain-v1
Ziya-LLaMA-13B-v1, Ziya-LLaMA-13B-v1.1
Baichuan
baichuan-inc
github, baichuan-inc/Baichuan-7B, baichuan-inc/Baichuan-13B-Base, baichuan-inc/Baichuan-13B-Chat
Baichuan-7B, Baichuan-13B-Base, Baichuan-13B-Chat
Baichuan2
baichuan-inc
github, baichuan-inc/Baichuan2-7B-Base, baichuan-inc/Baichuan2-7B-Chat, baichuan-inc/Baichuan2-13B-Base, baichuan-inc/Baichuan2-13B-Chat
Baichuan2-7B-Base, Baichuan2-7B-Chat, Baichuan2-13B-Base, Baichuan2-13B-Chat
vicuna
lmsys
lmsys/vicuna-7b-v1.5
vicuna-7b-v1.5
01-ai
github, 01-ai/Yi-6B, 01-ai/Yi-6B-200K
Yi-6B, Yi-6B-200K
bloom
bloom
bigscience
bigscience/bloom-560m, bigscience/bloomz-560m
bloom-560m, bloomz-560m
github, Qwen/Qwen-1_8B, Qwen/Qwen-1_8B-Chat, Qwen/Qwen-7B, Qwen/Qwen-7B-Chat
Qwen-1_8B, Qwen-1_8B-Chat, Qwen-7B, Qwen-7B-Chat
InternLM
InternLM
上海人工智能实验室
github, internlm/internlm-chat-7b, internlm/internlm-7b
internlm-7b, internlm-chat-7b
Falcon
Falcon
tiiuae
hf, tiiuae/falcon-rw-1b, tiiuae/falcon-7b, tiiuae/falcon-7b-instruct
falcon-rw-1b, falcon-7b, falcon-7b-instruct
deeoseek-moe
deepseek
github, deepseek-ai/deepseek-moe-16b-base, deepseek-ai/deepseek-moe-16b-chat
deepseek-moe-16b-base, deepseek-moe-16b-chat
embedding
text2vec-base-chinese
shibing624
shibing624/text2vec-base-chinese
text2vec-base-chinese
moka-ai
moka-ai/m3e-base
m3e-base
BAAI/bge-large-en-v1.5, BAAI/bge-large-zh-v1.5
bge-large-en-v1.5, bge-large-zh-v1.5
thenlper
thenlper/gte-large-zh, thenlper/gte-base-zh
gte-base-zh, gte-large-zh
高亮格式(如bert-base-chinese)的表示可直接build_transformer_model()联网下载
国内镜像网站加速下载
HF_ENDPOINT=https://hf-mirror.com python your_script.py
export HF_ENDPOINT=https://hf-mirror.com后再执行python代码
在python代码开头如下设置
import os
os.environ['HF_ENDPOINT'] = "https://hf-mirror.com"
6. 鸣谢
感谢苏神实现的bert4keras，本实现有不少地方参考了bert4keras的源码，在此衷心感谢大佬的无私奉献;
其次感谢项目bert4pytorch，也是在该项目的指引下给了我用pytorch来复现bert4keras的想法和思路。
7. 引用
@misc{bert4torch,
  title={bert4torch},
  author={Bo Li},
  year={2022},
  howpublished={\url{https://github.com/Tongjilibo/bert4torch}},
8. 其他
Wechat & Star History Chart


                   
                    
                     
                      
                      
                       
                        
                         FAQs
                        
                        
                         
                          What is bert4torch?
                         
                         
                          
                           
                            an elegant bert4torch. Visit Snyk Advisor to see a
                            
                             full health score report
                            
                            for bert4torch, including popularity, security, maintenance
            & community analysis.
                           
                          
                         
                        
                        
                         
                          Is bert4torch popular?
                         
                         
                          
                           
                            The python package bert4torch receives a total
              of
                            
                             667
                            
                            weekly downloads. As
              such, bert4torch popularity was classified as
                            
                             small
                            
                            . Visit the
                            
                             popularity section
                            
                            on Snyk Advisor to see the full health analysis.
                           
                          
                         
                        
                        
                        
                         
                          Is bert4torch well maintained?
                         
                         
                          
                           
                           
                           
                            We found that bert4torch demonstrated a
                            
                             healthy
                            
                            version release cadence and project
            activity. It has a community of
                            
                             4
                            
                            open source contributors
            collaborating on the project. See the full
                            
                             package health analysis
                            
                            to learn more about the package maintenance status.
                           
                          
                         
                        
                        
                         
                          Is bert4torch safe to use?
                         
                         
                          
                           
                           
                           
                            While scanning the latest version of bert4torch, we found
            that a
                            
                             security review is needed
                            
                            . A total of
            5 vulnerabilities or license issues were
            detected. See the full
                            
                             security scan results
                            
                            .
  Last updated on