|
1 | 1 | <?xml version="1.0" encoding="UTF-8" standalone="no"?> |
2 | | -<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /><title>17.2. 拆分文档</title><link rel="stylesheet" type="text/css" href="../../docbook.css" /><meta name="generator" content="DocBook XSL Stylesheets Vsnapshot" /><meta name="keywords" content="php,pear,pecl,phar, python, , " /><link rel="home" href="../../index.html" title="Netkiller Python 手札" /><link rel="up" href="index.html" title="第 17 章 LangChain" /><link rel="prev" href="index.html" title="第 17 章 LangChain" /><link rel="next" href="LangChain.OpenAI.html" title="17.3. ChatGPT" /></head><body><a xmlns="" href="//www.netkiller.cn/">Home</a> | <a xmlns="" href="//netkiller.github.io/">简体中文</a> | <a xmlns="" href="http://netkiller.sourceforge.net/">繁体中文</a> | <a xmlns="" href="/journal/index.html">杂文</a> |
3 | | - | <a xmlns="" href="https://github.com/netkiller">Github</a> | <a xmlns="" href="https://zhuanlan.zhihu.com/netkiller">知乎专栏</a> | <a xmlns="" href="https://www.facebook.com/bg7nyt">Facebook</a> | <a xmlns="" href="http://cn.linkedin.com/in/netkiller/">Linkedin</a> | <a xmlns="" href="https://www.youtube.com/user/bg7nyt/videos">Youtube</a> | <a xmlns="" href="//www.netkiller.cn/home/donations.html">打赏(Donations)</a> | <a xmlns="" href="//www.netkiller.cn/home/about.html">About</a><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">17.2. 拆分文档</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="index.html">上一页</a> </td><th width="60%" align="center">第 17 章 LangChain</th><td width="20%" align="right"> <a accesskey="n" href="LangChain.OpenAI.html">下一页</a></td></tr></table><hr /></div><table xmlns=""><tr><td><iframe src="//ghbtns.com/github-btn.html?user=netkiller&repo=netkiller.github.io&type=watch&count=true&size=large" height="30" width="170" frameborder="0" scrolling="0" style="width:170px; height: 30px;" allowTransparency="true"></iframe></td><td><iframe src="//ghbtns.com/github-btn.html?user=netkiller&repo=netkiller.github.io&type=fork&count=true&size=large" height="30" width="170" frameborder="0" scrolling="0" style="width:170px; height: 30px;" allowTransparency="true"></iframe></td><td><iframe src="//ghbtns.com/github-btn.html?user=netkiller&type=follow&count=true&size=large" height="30" width="240" frameborder="0" scrolling="0" style="width:240px; height: 30px;" allowTransparency="true"></iframe></td><td></td><td><a href="https://zhuanlan.zhihu.com/netkiller"><img src="/images/logo/zhihu-card-default.svg" height="25" /></a></td><td valign="middle"><a href="https://zhuanlan.zhihu.com/netkiller">知乎专栏</a></td><td></td><td></td><td></td><td></td></tr></table><div class="section"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a id="id3931"></a>17.2. 拆分文档</h2></div></div></div> |
| 2 | +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /><title>17.2. 拆分文档</title><link rel="stylesheet" type="text/css" href="../docbook.css" /><meta name="generator" content="DocBook XSL Stylesheets Vsnapshot" /><meta name="keywords" content="php,pear,pecl,phar, python, , " /><link rel="home" href="../index.html" title="Netkiller Python 手札" /><link rel="up" href="index.html" title="第 17 章 LangChain" /><link rel="prev" href="index.html" title="第 17 章 LangChain" /><link rel="next" href="LangChain.OpenAI.html" title="17.3. ChatGPT" /></head><body><a xmlns="" href="//www.netkiller.cn/">Home</a> | <a xmlns="" href="//netkiller.github.io/">简体中文</a> | <a xmlns="" href="http://netkiller.sourceforge.net/">繁体中文</a> | <a xmlns="" href="/journal/index.html">杂文</a> |
| 3 | + | <a xmlns="" href="https://github.com/netkiller">Github</a> | <a xmlns="" href="https://zhuanlan.zhihu.com/netkiller">知乎专栏</a> | <a xmlns="" href="https://www.facebook.com/bg7nyt">Facebook</a> | <a xmlns="" href="http://cn.linkedin.com/in/netkiller/">Linkedin</a> | <a xmlns="" href="https://www.youtube.com/user/bg7nyt/videos">Youtube</a> | <a xmlns="" href="//www.netkiller.cn/home/donations.html">打赏(Donations)</a> | <a xmlns="" href="//www.netkiller.cn/home/about.html">About</a><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">17.2. 拆分文档</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="index.html">上一页</a> </td><th width="60%" align="center">第 17 章 LangChain</th><td width="20%" align="right"> <a accesskey="n" href="LangChain.OpenAI.html">下一页</a></td></tr></table><hr /></div><table xmlns=""><tr><td><iframe src="//ghbtns.com/github-btn.html?user=netkiller&repo=netkiller.github.io&type=watch&count=true&size=large" height="30" width="170" frameborder="0" scrolling="0" style="width:170px; height: 30px;" allowTransparency="true"></iframe></td><td><iframe src="//ghbtns.com/github-btn.html?user=netkiller&repo=netkiller.github.io&type=fork&count=true&size=large" height="30" width="170" frameborder="0" scrolling="0" style="width:170px; height: 30px;" allowTransparency="true"></iframe></td><td><iframe src="//ghbtns.com/github-btn.html?user=netkiller&type=follow&count=true&size=large" height="30" width="240" frameborder="0" scrolling="0" style="width:240px; height: 30px;" allowTransparency="true"></iframe></td><td></td><td><a href="https://zhuanlan.zhihu.com/netkiller"><img src="/images/logo/zhihu-card-default.svg" height="25" /></a></td><td valign="middle"><a href="https://zhuanlan.zhihu.com/netkiller">知乎专栏</a></td><td></td><td></td><td></td><td></td></tr></table><div class="section"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a id="id3977"></a>17.2. 拆分文档</h2></div></div></div> |
4 | 4 |
|
5 | | - <div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="id1138"></a>17.2.1. 拆分文本</h3></div></div></div> |
| 5 | + <div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="id1139"></a>17.2.1. 拆分文本</h3></div></div></div> |
6 | 6 |
|
7 | 7 | <pre class="programlisting"> |
8 | 8 |
|
|
29 | 29 |
|
30 | 30 | </pre> |
31 | 31 | </div> |
32 | | - <div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="id1139"></a>17.2.2. 拆分代码</h3></div></div></div> |
| 32 | + <div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="id1140"></a>17.2.2. 拆分代码</h3></div></div></div> |
33 | 33 |
|
34 | 34 | <pre class="programlisting"> |
35 | 35 |
|
|
48 | 48 |
|
49 | 49 | </pre> |
50 | 50 | </div> |
51 | | - <div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="id1140"></a>17.2.3. 拆分 Markdown 文档</h3></div></div></div> |
| 51 | + <div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="id1141"></a>17.2.3. 拆分 Markdown 文档</h3></div></div></div> |
52 | 52 |
|
53 | 53 | <pre class="programlisting"> |
54 | 54 |
|
|
65 | 65 |
|
66 | 66 | </pre> |
67 | 67 | </div> |
68 | | - <div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="id1141"></a>17.2.4. 按token拆分</h3></div></div></div> |
| 68 | + <div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="id1142"></a>17.2.4. 按token拆分</h3></div></div></div> |
69 | 69 |
|
70 | 70 | <p>例如 OpenAI 的 token 有字数限制。在API调用时不应超过 token 限制，使用 from_tiktoken_encoder 可以解决这个问题。</p> |
71 | 71 | <pre class="programlisting"> |
|
88 | 88 |
|
89 | 89 | </pre> |
90 | 90 | </div> |
91 | | - </div><script xmlns="" type="text/javascript" id="clustrmaps" src="//cdn.clustrmaps.com/map_v2.js?u=r5HG&d=9mi5r_kkDC8uxG8HuY3p4-2qgeeVypAK9vMD-2P6BYM"></script><div class="navfooter"><hr /><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="index.html">上一页</a> </td><td width="20%" align="center"><a accesskey="u" href="index.html">上一级</a></td><td width="40%" align="right"> <a accesskey="n" href="LangChain.OpenAI.html">下一页</a></td></tr><tr><td width="40%" align="left" valign="top">第 17 章 LangChain </td><td width="20%" align="center"><a accesskey="h" href="../../index.html">起始页</a></td><td width="40%" align="right" valign="top"> 17.3. ChatGPT</td></tr></table></div><script xmlns=""> |
| 91 | + </div><script xmlns="" type="text/javascript" id="clustrmaps" src="//cdn.clustrmaps.com/map_v2.js?u=r5HG&d=9mi5r_kkDC8uxG8HuY3p4-2qgeeVypAK9vMD-2P6BYM"></script><div class="navfooter"><hr /><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="index.html">上一页</a> </td><td width="20%" align="center"><a accesskey="u" href="index.html">上一级</a></td><td width="40%" align="right"> <a accesskey="n" href="LangChain.OpenAI.html">下一页</a></td></tr><tr><td width="40%" align="left" valign="top">第 17 章 LangChain </td><td width="20%" align="center"><a accesskey="h" href="../index.html">起始页</a></td><td width="40%" align="right" valign="top"> 17.3. ChatGPT</td></tr></table></div><script xmlns=""> |
92 | 92 | (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ |
93 | 93 | (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), |
94 | 94 | m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) |
|
0 commit comments