search.xml

<?xml version="1.0" encoding="utf-8"?>
<search>
  
  
    
    <entry>
      <title><![CDATA[Notes on High Performance MySQL]]></title>
      <url>https://blog.staynoob.cn/post/2024/06/notes-on-high-performance-mysql/</url>
      <content type="html"><![CDATA[<blockquote>
<p>Notes on <a href="https://www.amazon.com/High-Performance-MySQL-Optimization-Replication/dp/1449314287/ref=sr_1_5?keywords=mysql&amp;qid=1560589416&amp;s=books&amp;sr=1-5">High Performance MySQL</a>, for future reference.</p>
</blockquote>
<h3 id="Chapter-4-Optimizing-Schema-and-Data-Types"><a href="#Chapter-4-Optimizing-Schema-and-Data-Types" class="headerlink" title="Chapter 4: Optimizing Schema and Data Types"></a>Chapter 4: Optimizing Schema and Data Types</h3><h4 id="Choosing-Identifiers"><a href="#Choosing-Identifiers" class="headerlink" title="Choosing Identifiers"></a>Choosing Identifiers</h4><blockquote>
<p>Integers are usually the best choice of identifiers. Avoid string types for identifiers if possible, because they take up a lot of space and are generally slower than integer types. You should also be very careful with completely “random” strings, such as those produced by MD5() , SHA1() , or UUID().</p>
</blockquote>
<span id="more"></span>
<h4 id="Datetime-vs-Timestamp"><a href="#Datetime-vs-Timestamp" class="headerlink" title="Datetime vs Timestamp"></a>Datetime vs Timestamp</h4><table>
<thead>
<tr>
<th>type</th>
<th>size</th>
<th>date range</th>
<th>display depends on timezone?</th>
</tr>
</thead>
<tbody>
<tr>
<td>datetime</td>
<td>8 byte</td>
<td>1001-9999</td>
<td>no</td>
</tr>
<tr>
<td>timestamp</td>
<td>4 byte</td>
<td>1970-2038</td>
<td>yes</td>
</tr>
</tbody>
</table>
<blockquote>
<p>Special behavior aside, in general if you can use TIMESTAMP you should, because it is<br>more space-efficient than DATETIME . Sometimes people store Unix timestamps as integer<br>values, but this usually doesn’t gain you anything. The integer format is often less<br>convenient to deal with, so we do not recommend doing this.</p>
</blockquote>
<h4 id="Avoid-NULL-if-possible"><a href="#Avoid-NULL-if-possible" class="headerlink" title="Avoid NULL if possible"></a>Avoid NULL if possible</h4><blockquote>
<p>We suggest considering alternatives when possible. Even when you do need to store a &quot;no value&quot; fact in a table, you might not need to use NULL. Perhaps you can use zero, a special value, or an empty string instead. However, don’t be too afraid of using NULL when you need to represent an unknown value. In some cases, it’s better to use NULL than a magical constant. Selecting one value from the domain of a constrained type, such as using −1 to represent an unknown integer, can complicate your code a lot, introduce bugs, and just generally make a total mess out of things. Handling NULL isn&#39;t always easy, but it’s often better than the alternative.</p>
</blockquote>
<p>However, in my point of view, if you need to store a &quot;no value&quot; fact, NULL is always the best choice, every programmer understand its meaning, the little performance improvements(by avoid null value) should not be taken into consideration until it really bothers you, besides, the fact that your default value take less space and has better performance is highly depends on implementation detail of the store engine. It&#39;s hard to conclude that it will be always true in future releases, let alone other store engines and other database products.</p>
<h3 id="Chapter5-Indexing-for-high-performance"><a href="#Chapter5-Indexing-for-high-performance" class="headerlink" title="Chapter5: Indexing for high performance"></a>Chapter5: Indexing for high performance</h3><h4 id="Types-of-queries-that-can-use-a-B-Tree-index"><a href="#Types-of-queries-that-can-use-a-B-Tree-index" class="headerlink" title="Types of queries that can use a B-Tree index."></a>Types of queries that can use a B-Tree index.</h4><blockquote>
<p>Suppose we have the following table:<br><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">CREATE</span> <span class="keyword">TABLE</span> People (</span><br><span class="line">    last_name <span class="type">varchar</span>(<span class="number">50</span>) <span class="keyword">not</span> <span class="keyword">null</span>,</span><br><span class="line">    first_name <span class="type">varchar</span>(<span class="number">50</span>) <span class="keyword">not</span> <span class="keyword">null</span>,</span><br><span class="line">    dob <span class="type">date</span> <span class="keyword">not</span> <span class="keyword">null</span>,</span><br><span class="line">    gender enum(<span class="string">&#x27;m&#x27;</span>, <span class="string">&#x27;f&#x27;</span>)<span class="keyword">not</span> <span class="keyword">null</span>,</span><br><span class="line">    key(last_name, first_name, dob)</span><br><span class="line">);</span><br></pre></td></tr></table></figure><br>The index will be useful for the following kinds of queries:</p>
<ul>
<li>Match the full value<br>  A match on the full key value specifies values for all columns in the index. For example, this index can help you find a person named Cuba Allen who was born on 1960-01-01.</li>
<li><strong> Match a leftmost prefix </strong><br>  This index can help you find all people with the last name Allen. This uses only the first column in the index.</li>
<li><strong> Match a column prefix </strong><br>  You can match on the first part of a column&#39;s value. This index can help you find all people whose last names begin with J. This uses only the first column in the index.</li>
<li>Match a range of values<br>  This index can help you find people whose last names are between Allen and Barrymore. This also uses only the first column.</li>
<li>Match one part exactly and match a range on another part<br>  This index can help you find everyone whose last name is Allen and whose first name starts with the letter K (Kim, Karl, etc.). This is an exact match on <code>last_name</code> and a range query on <code>first_name</code> .</li>
<li>Index-only queries<br>  B-Tree indexes can normally support index-only queries, which are queries that access only the index, not the row storage. We discuss this optimization in &quot;Covering Indexes&quot; on page 177.</li>
</ul>
</blockquote>
<blockquote>
<p>Here are some limitations of B-Tree indexes:</p>
<ul>
<li>They are not useful if the lookup does not start from the leftmost side of the indexed columns. For example, this index won&#39;t help you find all people named Bill or all people born on a certain date, because those columns are not leftmost in the index.  Likewise, you can&#39;t use the index to find people whose last name ends with a particular letter.</li>
<li>You can&#39;t skip columns in the index. That is, you won&#39;t be able to find all people whose last name is Smith and who were born on a particular date. If you don&#39;t specify a value for the <code>first_name</code> column, MySQL can use only the first column of the index.</li>
</ul>
</blockquote>
]]></content>
      
        <categories>
            
            <category> Backend </category>
            
        </categories>
        
        
        <tags>
            
            <tag> Database </tag>
            
            <tag> SQL </tag>
            
        </tags>
        
    </entry>
    
    <entry>
      <title><![CDATA[Exploration vs Exploitation]]></title>
      <url>https://blog.staynoob.cn/post/2024/06/exploration-vs-exploitation/</url>
      <content type="html"><![CDATA[<blockquote>
<p>A short note on <a href="https://youtu.be/FgzM3zpZ55o?si=UdTA9MaP9EalwMu5&amp;t=3311">Standford CS234 Reinforcement Learning 2019 Lecture1</a><br>How should an RL agent balance its action?</p>
<ul>
<li>Exploration: trying new things that might enable the agent to make better decisions in the future</li>
<li>Exploitation: choosing actions that are expected to yield good reward given the past experience</li>
</ul>
<p>Often there may be an exploration-exploitation tradeoff, we may have to sacrifice reward in order to explore and learn about better policy.</p>
</blockquote>
<p>To make the idea concrete, if you go to a restaurant, they have several different dishes, you want to optimize at the best dish, the best strategy is actually depends on how long you will spend near that restaurant. If you are going to live there for a long time, the best strategy is try them all, instead, when you go to the restaurant last time, you should order the known best dish.<br>The underlying idea is fairly simple, when it applys to human lives, it means you should try different things while you are young, and stick to whatever interests you when you gets old.<br>It also suggests, &quot;Treat everyday as if it&#39;s your last day&quot; is actually a terrible strategy. Because if it is your last day, you should always choose to do whatever gives you the maximum pleasure, but if you have future, you should take more time for &quot;exploration&quot;.</p>
]]></content>
      
        <categories>
            
            <category> Diary </category>
            
        </categories>
        
        
    </entry>
    
    <entry>
      <title><![CDATA[Why I think Kotlin is preferable to Java]]></title>
      <url>https://blog.staynoob.cn/post/2019/10/why-i-think-kotlin-is-preferable-to-java/</url>
      <content type="html"><![CDATA[<blockquote>
<p>Although I&#39;m quite impressed by Rust language recently, Kotlin is still my favorite language. In this post, I will share the major reasons which convinced me to leave Java two years ago. It won&#39;t cover every bright side of Kotlin language, but will be enough to make my point.</p>
</blockquote>
<p>TL:DR</p>
<h3 id="Java-the-Good-Parts-and-the-Bad-Parts"><a href="#Java-the-Good-Parts-and-the-Bad-Parts" class="headerlink" title="Java the Good Parts and the Bad Parts"></a>Java the Good Parts and the Bad Parts</h3><p>If you ever asked me if Java is a good programming language, I would definitely say yes. Compare to languages such as C++, VB, Javascript. Writing code in Java is much more pleasant. More specifically, its virtue including but not limited to:</p>
<ul>
<li>Cross Platform</li>
<li>Statically Typed</li>
<li>Automatic Memory Management</li>
<li>Open Community</li>
<li>(After all, When I could not make a living by writing some fancy languages. It was Java gave me a job so that I could complain it all day.)</li>
</ul>
<p>Anyway, just like other elder languages, Java has made many design mistakes, I won&#39;t dive into the language design topic here, as I&#39;m not a specialist in programming language (or any other) field. I just want to share some issues that do bother me, from a mediocre programmer&#39;s perspective, then see how they are solved in Kotlin.</p>
<span id="more"></span>
<h4 id="Null-safety"><a href="#Null-safety" class="headerlink" title="Null safety"></a>Null safety</h4><p>The first problem is the notorious Null reference(aka <a href="https://en.wikipedia.org/wiki/Tony_Hoare#Apologies_and_retractions">The billion-dollar mistake</a>), to people who don&#39;t understand why it is a design mistake, considering the following example, suppose some libraries author wrote a method that returns a User&#39;s full name.<br><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">public</span> <span class="keyword">interface</span> <span class="title class_">User</span> &#123;</span><br><span class="line">    String <span class="title function_">getFullName</span><span class="params">()</span>;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br>then you want to write a function which tests if a user is John Snow, you may end up with writing code like this:<br><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">public</span> <span class="type">boolean</span> <span class="title function_">isJohnSnow</span><span class="params">(User user)</span> &#123;</span><br><span class="line">    <span class="keyword">return</span> user.getFullName().equals(<span class="string">&quot;John Snow&quot;</span>);</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br>As a Java novice, You test your code and it works smoothly, then you push it to production. Now, Your workmates are able to make use of it. They may want to say something to John.<br><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title function_">greeting</span><span class="params">(User user)</span> &#123;</span><br><span class="line">    <span class="keyword">if</span> (isJohnSnow(user)) &#123;</span><br><span class="line">        sendMessage(user, <span class="string">&quot;You know nothing!&quot;</span>);</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br>Someday, Your workmates complain that your method throws an NPE, because no one says the <code>getFullName()</code> method is not allowed to return a null, which indicates that they don&#39;t know the user&#39;s full name. So you take the blame and say sorry to your workmates, then you fix your code immediately:<br><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">public</span> Boolean <span class="title function_">isJohnSnow</span><span class="params">(User user)</span> &#123;</span><br><span class="line">    <span class="type">String</span> <span class="variable">fullName</span> <span class="operator">=</span> user.getFullName();</span><br><span class="line">    <span class="keyword">if</span> (fullName == <span class="literal">null</span>) <span class="keyword">return</span> <span class="literal">null</span>;</span><br><span class="line">    <span class="keyword">return</span> fullName.equals(<span class="string">&quot;John Snow&quot;</span>);</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br>You change the method signature to return the boxed type and return a null reference indicates you don&#39;t know if the user is John Snow. Thanks to auto unboxing, none of your workmates will notice this change, until the day their code starts to throw NPE...<br>So how do we avoid this problem in Java? We do have some options:</p>
<ul>
<li>Check before calling<br>A naive way would be whenever you call a method, you check its implementation and find out if the method will return null. It is naive because the author of the method can change its implementation at any time. Also, in the preceding case, you may have to check every class which implements a User interface, this is not really practical. Moreover, the whole point of OOP and statically typed is you don&#39;t need to care about implementations, in most cases the signature of a method will tell you enough information.</li>
<li>Check after returning<br>Another possible option is being extremely pessimistic, which means you cannot trust every object returned by other methods, although most of them never return a null reference, you still have to write <code>if(returned == null)</code> everywhere, because you never read their documents and their signature never says return null is not allowed. However, as you can guess, this is not practical either, because writing and reading such tedious code drives people crazy.</li>
<li>Return a meaningful default value in place of null.<br>As NPE is such an annoying thing, you may think we can avoid it by not using Null reference. For example, we can return an empty String to indicate we don&#39;t know a user&#39;s full name. Again it&#39;s not a good idea, empty string and &quot;missing value&quot; are not the same. <strong> In fact, if you want to express a &quot;missing&quot; value, there is no other thing better than a null reference, every programmer can understand its meaning.</strong> After all, I never said It is Null&#39;s fault to cause a NullPointerException.</li>
<li>Optional<br>If your method sometimes returns a &quot;missing value&quot;, use Optional as the return type is a very good option. However, it doesn&#39;t solve all the problems because an Optional itself is a reference type, which means it could be a null reference.</li>
<li>Nullable annotation<br>Another good option to reduce the risk of NPE is to use an annotation that indicates your method may return a null value. For example:<figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">public</span> <span class="keyword">interface</span> <span class="title class_">User</span> &#123;</span><br><span class="line">    <span class="meta">@Nullable</span></span><br><span class="line">    String <span class="title function_">getFullName</span><span class="params">()</span>;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure>
Then the IDEs such as Idea can warn you when you forget to handle the possible null value. However, you are not guaranteed to get the benefit from doing this when you switch to another IDE.</li>
</ul>
<p>As you can see, there is no perfect solution to avoid the annoying NPE. Before introducing how modern languages(Kotlin, Swift, Typescript in StrictNull model) solve this problem, let&#39;s think about why Tony Hoare said this is a mistake. In my point of view, it&#39;s a quintessential example of violating the <a href="https://en.wikipedia.org/wiki/Liskov_substitution_principle">Liskov substitution principle</a> (However, LSP is almost twenty years later than Hoare invented Null reference). Namely, null is a subtype of any other reference type in Java, but it doesn&#39;t have their behavior.<br><img src="/img/content/why-i-think-kotlin-is-preferable-to-java/java-null-relation.png" alt="java-null-relation"><br>More specifically, in our preceding example, you can see that the Java compiler allow the null reference to go anywhere a String can go, but it doesn&#39;t have the String&#39;s behavior, it doesn&#39;t support <code>equals</code>, <code>startWith</code> operation. That&#39;s why we get NPE.<br>On the contrary, In modern languages like Kotlin. The relation between Nullable String, String, null is depicted as follows.<br><img src="/img/content/why-i-think-kotlin-is-preferable-to-java/kotlin-null-relation.png" alt="kotlin-null-relation"><br>String and null are both the subtype of nullable String Type(in Kotlin it is denoted as <code>String?</code>), but null is not a String anymore, which means String and null can go anywhere a <code>String?</code> can go. but if your method signature says you are returning a String value, the compiler will stop you from returning a null. Thus the caller of your method can safely call String&#39;s methods on the returned value without checking the value first. And if you claim that your method will return a Nullable String? The compiler will force the caller to check the returned value before any further processing. Moreover, Kotlin also provides several operators to help you handle Nullable type easily and safely, such like <code>?.</code>, <code>?:</code>, <code>as?</code>, Check out the <a href="https://kotlinlang.org/docs/reference/null-safety.html">official documentation</a> to get more information about them.</p>
<h4 id="Immutable-Collection"><a href="#Immutable-Collection" class="headerlink" title="Immutable Collection"></a>Immutable Collection</h4><p>Another example of violating the Liskov&#39;s principle is Java collection API. Here is my story, when I was a beginner, I found that <code>Arrays.asList</code> comes very handy, so I used to write something like:<br><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">List&lt;String&gt; list = Arrays.asList(<span class="string">&quot;foo&quot;</span>, <span class="string">&quot;bar&quot;</span>);</span><br></pre></td></tr></table></figure><br>To ensure everything is ok, I even checked its source code.<br><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// source code of Arrays.asList</span></span><br><span class="line"><span class="meta">@SafeVarargs</span></span><br><span class="line"><span class="meta">@SuppressWarnings(&quot;varargs&quot;)</span></span><br><span class="line"><span class="keyword">public</span> <span class="keyword">static</span> &lt;T&gt; List&lt;T&gt; <span class="title function_">asList</span><span class="params">(T... a)</span> &#123;</span><br><span class="line">    <span class="keyword">return</span> <span class="keyword">new</span> <span class="title class_">ArrayList</span>&lt;&gt;(a);</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br>As I expected, it returns my favorite ArrayList. Later somewhere, I add an element into the list as usual.<br><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">list.add(<span class="string">&quot;baz&quot;</span>);</span><br></pre></td></tr></table></figure><br>So what could possibly go wrong? I can still remember how shocked I was when I found out it throws a <code>UnsupportedOperationException</code> at runtime, turns out the <code>ArrayList</code> is just a private class defined in Arrays.java, it doesn&#39;t support add operation.<br>In Java, for some historical reason (I guess), <code>UnmodifiableCollection</code> is a subtype of Collection. So are <code>UnmodifiableSet</code>, <code>UnmodifiableList</code> etc. which means if you get a List returned by other people, you simply don&#39;t know if you can add an element into it. Just like the previous NPE problem, you can either read the documentation if there is one, or go through the source code to find out its actual return type. Again, these options are not practical for exactly the same reason, you cannot read the documentation or source code before every method call.<br>Luckily, mutate a List which is not created by yourself is not a good practice, so I tend to ignore the mutate operation in List interface, and treat every List instance as Immutable.<br>In Kotlin, <code>Collection</code> and <code>MutableCollection</code> are separated interface, more precisely, <code>MutableCollection</code> is a subtype of <code>Collection</code>. If you have a <code>Collection</code> interface, the compiler won&#39;t let you mutate it, If you have a <code>MutableCollection</code> instance, you are promised that you can mutate it, if your method expects a <code>Collection</code>, you might get a <code>MutableCollection</code>, but not vice versa, everything is simple and clear.</p>
<h4 id="Collection-Operation"><a href="#Collection-Operation" class="headerlink" title="Collection Operation"></a>Collection Operation</h4><p>Java8 introduced the Stream API, It allows people manipulate collections in a declarative way, which means you only need to specify what you need, not how you do, leave the implementation detail to the framework. Also, thanks to its lazy fact, a lot of optimization can be done under the hood. Furthermore, you can even change your stream to &quot;parallel&quot; mode any time you want, how awesome it is! But let me ask you a question, what was the last time you change your stream to parallel?<br>For me, I never do this. Most time I&#39;m dealing with a collection contains no more than 1000 elements, I don&#39;t really care if it could be optimized a little bit, it doesn&#39;t make observable difference anyway. Besides, If somehow my collection grows to millions of elements. I don&#39;t think the underlying optimization can save me from rewrite my code, neither switching to parallel mode will do.<br>While I can&#39;t see what it is good for, I do see its drawback. It complicates things a lot. That is, if you are a programmer who has already been familiar with languages such as Javascript, Python. Highly likely you are still unable to figure out how <code>map</code>, <code>filter</code>, <code>reduce</code> are done in Java without reading the f* manual. Due to the Stream, Collector class make heavily use of overload, generic methods, you usually cannot get useful hints when you stuck at some point, the compiler may produce inscrutable, even completed irrelevant error message.<br>Even though you did everything right, the API still has its own limit. For example, If I need to get the index of current iterating element, I have no choice but rewrite the whole method calling chain back to the old &quot;for loop&quot; form. You may argue the following code could do it the stream way.<br><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">IntStream.range(<span class="number">0</span>, list.size())</span><br><span class="line">        .boxed()</span><br><span class="line">        .collect(Collectors.toMap(</span><br><span class="line">                e -&gt; e,     <span class="comment">// index</span></span><br><span class="line">                list::get   <span class="comment">// element</span></span><br><span class="line">        ));</span><br></pre></td></tr></table></figure><br>However, It&#39;s not acceptable to me. Not to mention the code hides the intention, its performance may also be a problem, as you can see, it takes N^2 time to traverse over a <code>LinkedList</code>.<br>Altogether, It gives me the sense that the authors of the API just have to be so smart, inventing something simple is insulting their intelligence.<br>In contrast, most collection operation in Kotlin is implemented by inline function, here is the source code of <code>mapIndex</code> method.<br><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">public</span> inline fun &lt;T, R&gt; Iterable&lt;T&gt;.mapIndexed(transform: (index: Int, T) -&gt; R): List&lt;R&gt; &#123;</span><br><span class="line">    <span class="keyword">return</span> mapIndexedTo(ArrayList&lt;R&gt;(collectionSizeOrDefault(<span class="number">10</span>)), transform)</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="keyword">public</span> inline fun &lt;T, R, C : MutableCollection&lt;in R&gt;&gt; Iterable&lt;T&gt;.mapIndexedTo(destination: C, transform: (index: Int, T) -&gt; R): C &#123;</span><br><span class="line">    <span class="type">var</span> <span class="variable">index</span> <span class="operator">=</span> <span class="number">0</span></span><br><span class="line">    <span class="keyword">for</span> (item in <span class="built_in">this</span>)</span><br><span class="line">        destination.add(transform(index++, item))</span><br><span class="line">    <span class="keyword">return</span> destination</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br>The inline modifier indicates the implementation will be inline to the call site so that it won&#39;t introduce extra runtime overhead. In other words, it works just like a kind of syntactic sugar. Later, if you find you do need the stream behavior, thanks to the compatibility between Kotlin and Java, the stream API is still there for you.<br>With Kotlin type induction and the powerful Idea, you can peek your result type on each step, therefore it is easy to figure out what is going wrong, here is a simple example shows how collection operation in Kotlin looks like.</p>
<p><img src="/img/content/why-i-think-kotlin-is-preferable-to-java/kotlin-group-by.gif" alt="kotlin-group-by"></p>
<h4 id="Template-Code"><a href="#Template-Code" class="headerlink" title="Template Code"></a>Template Code</h4><p>Java is doing an excellent job if you get paid by counting your code lines. If you want to get rich, all you need to do is create some so called &quot;POJO&quot;, add some fields. Here is an example, basically just 13 lines of code.<br><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">public</span> <span class="keyword">class</span> <span class="title class_">Foo</span> &#123;</span><br><span class="line"></span><br><span class="line">    <span class="keyword">private</span> String field1;</span><br><span class="line">    <span class="keyword">private</span> String field2;</span><br><span class="line">    <span class="keyword">private</span> String field3;</span><br><span class="line"></span><br><span class="line">    <span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">class</span> <span class="title class_">Bar</span> &#123;</span><br><span class="line"></span><br><span class="line">        <span class="keyword">private</span> String field1;</span><br><span class="line">        <span class="keyword">private</span> String field2;</span><br><span class="line">        <span class="keyword">private</span> String field3;</span><br><span class="line"></span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    <span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">class</span> <span class="title class_">Baz</span> &#123;</span><br><span class="line"></span><br><span class="line">        <span class="keyword">private</span> String field1;</span><br><span class="line">        <span class="keyword">private</span> String field2;</span><br><span class="line">        <span class="keyword">private</span> String field3;</span><br><span class="line"></span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br>Then, let IDE help finish your job. When it gets done, it automatically becomes hundreds of lines.<br><img src="/img/content/why-i-think-kotlin-is-preferable-to-java/java-template-code.png" alt="java-template-code"><br>How amazing it is!<br>Sadly, in Kotlin, it will still be 13 lines.<br><figure class="highlight kotlin"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">data</span> <span class="keyword">class</span> <span class="title class_">Foo</span>(</span><br><span class="line">    <span class="keyword">var</span> field1: String,</span><br><span class="line">    <span class="keyword">var</span> field2: String,</span><br><span class="line">    <span class="keyword">var</span> field3: String</span><br><span class="line">) &#123;</span><br><span class="line"></span><br><span class="line">    <span class="keyword">data</span> <span class="keyword">class</span> <span class="title class_">Bar</span>(</span><br><span class="line">        <span class="keyword">var</span> field1: String,</span><br><span class="line">        <span class="keyword">var</span> field2: String,</span><br><span class="line">        <span class="keyword">var</span> field3: String</span><br><span class="line">    )</span><br><span class="line"></span><br><span class="line">    <span class="keyword">data</span> <span class="keyword">class</span> <span class="title class_">Baz</span>(</span><br><span class="line">        <span class="keyword">var</span> field1: String,</span><br><span class="line">        <span class="keyword">var</span> field2: String,</span><br><span class="line">        <span class="keyword">var</span> field3: String</span><br><span class="line">    )</span><br><span class="line"></span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br>The <code>data</code> keyword covers all the <code>equals</code>, <code>hashcode</code>,<code>toString</code> functionality, and even more powerful.</p>
<h3 id="Effective-Java"><a href="#Effective-Java" class="headerlink" title="Effective Java"></a>Effective Java</h3><p>Effective Java(we refer to edition 2 here) is a great book in Java field, it helps me avoiding tons of pitfall in Java, in this section, you will see how Kotlin adhere its advice.</p>
<h4 id="Singleton-Item-3-Enforce-the-singleton-property-with-a-private-constructor-or-an-enum-type"><a href="#Singleton-Item-3-Enforce-the-singleton-property-with-a-private-constructor-or-an-enum-type" class="headerlink" title="Singleton (Item 3: Enforce the singleton property with a private constructor or an enum type)"></a>Singleton (Item 3: Enforce the singleton property with a private constructor or an enum type)</h4><p>I have been asked how to implement a Singleton once in an interview, because it is not trivial in Java (Considering the <code>setAccessible</code>, <code>Serializable</code> factor). As the book suggests, The best way to implement a singleton is to use a single-element enum type.<br><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">public</span> <span class="keyword">enum</span> <span class="title class_">Elvis</span> &#123;</span><br><span class="line">    INSTANCE;</span><br><span class="line"></span><br><span class="line">    <span class="comment">// properties</span></span><br><span class="line">    <span class="comment">// methods</span></span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br>However, I think this approach is somehow hacky, the <code>enum</code> keyword is not designed for this purpose, no one will understand why use <code>enum</code> here at first glance, hence it does make it a fair interview question.<br>Sadly again, you cannot ask a Kotlin programmer how to implement a Singleton because it is so damn easy and obvious.<br><figure class="highlight kotlin"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">object</span> Elvis &#123;</span><br><span class="line"></span><br><span class="line">    <span class="comment">// properties</span></span><br><span class="line">    <span class="comment">// methods</span></span><br><span class="line"></span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure></p>
<h4 id="Immutability-Item-15-Minimize-mutability"><a href="#Immutability-Item-15-Minimize-mutability" class="headerlink" title="Immutability (Item 15: Minimize mutability)"></a>Immutability (Item 15: Minimize mutability)</h4><p>I will not count the benefits of making object immutable here. Even a Javascript(a single-threaded language) programmer will know its importance(see <a href="https://redux.js.org/faq/immutable-data">Redux</a> and <a href="https://github.com/immutable-js/immutable-js">Immutable.js</a>). Not to mention the situation in Java.<br>Nonetheless, it really cost a lot to make a Java POJO immutable. You may argue how hard it could be? Just go make everything final. Let&#39;s see, suppose we have a &quot;POJO&quot; which have five fields.<br><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">public</span> <span class="keyword">final</span> <span class="keyword">class</span> <span class="title class_">Sample</span> &#123;</span><br><span class="line"></span><br><span class="line">    <span class="keyword">private</span> <span class="keyword">final</span> String field1;</span><br><span class="line">    <span class="keyword">private</span> <span class="keyword">final</span> Boolean field2;</span><br><span class="line">    <span class="keyword">private</span> <span class="keyword">final</span> Integer field3;</span><br><span class="line">    <span class="keyword">private</span> <span class="keyword">final</span> Float field4;</span><br><span class="line">    <span class="keyword">private</span> <span class="keyword">final</span> Double field5;</span><br><span class="line"></span><br><span class="line">    <span class="keyword">public</span> <span class="title function_">Sample</span><span class="params">(String field1, Boolean field2, Integer field3, Float field4, Double field5)</span> &#123;</span><br><span class="line">        <span class="built_in">this</span>.field1 = field1;</span><br><span class="line">        <span class="built_in">this</span>.field2 = field2;</span><br><span class="line">        <span class="built_in">this</span>.field3 = field3;</span><br><span class="line">        <span class="built_in">this</span>.field4 = field4;</span><br><span class="line">        <span class="built_in">this</span>.field5 = field5;</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    <span class="comment">// getter is omitted</span></span><br><span class="line"></span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br>Admittedly, create such a class is not hard, but work with it could bugs your head out. Create such an instance will like<br><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="type">Sample</span> <span class="variable">sample1</span> <span class="operator">=</span> <span class="keyword">new</span> <span class="title class_">Sample</span>(<span class="string">&quot;&quot;</span>, <span class="literal">true</span>, <span class="number">0</span>, <span class="number">0f</span>, <span class="number">0d</span>);</span><br></pre></td></tr></table></figure><br>I promise you that no one will understand the meaning of each parameter include yourself. Suppose you want to change the <code>field2</code> and create a new instance it will be like<br><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="type">Sample</span> <span class="variable">sample2</span> <span class="operator">=</span> <span class="keyword">new</span> <span class="title class_">Sample</span>(sample1.getField1(), <span class="literal">false</span>, sample1.getField3(), sample1.getField4(), sample1.getField5());</span><br></pre></td></tr></table></figure><br>If this is acceptable to you, I sincerely wish you never need to add a new field to that class. I know this kind of problem can be solved by following the builder pattern, but most programmers are lazy, they won&#39;t write the code until the day they have to. That is, <strong> in the ideal Java world, we should make a class immutable unless we have a good reason not to do so. In practical, we leave a class mutable unless we know it will cause problems in advance. </strong><br>In Kotlin, create and work with an immutable classes is even easier than the mutable one.<br><figure class="highlight kotlin"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">data</span> <span class="keyword">class</span> <span class="title class_">Sample</span>(</span><br><span class="line">    <span class="keyword">val</span> field1: String = <span class="string">&quot;&quot;</span>,</span><br><span class="line">    <span class="keyword">val</span> field2: <span class="built_in">Boolean</span> = <span class="literal">false</span>,</span><br><span class="line">    <span class="keyword">val</span> field3: <span class="built_in">Int</span> = <span class="number">0</span>,</span><br><span class="line">    <span class="keyword">val</span> field4: <span class="built_in">Float</span> = <span class="number">0f</span>,</span><br><span class="line">    <span class="keyword">val</span> field5: <span class="built_in">Double</span>? = <span class="literal">null</span></span><br><span class="line">)</span><br></pre></td></tr></table></figure><br>To &quot;minimize mutability&quot;, Kotlin class are final by default, with default parameter and naming parameter, you can do something like.<br><figure class="highlight kotlin"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// they are all valid</span></span><br><span class="line"><span class="keyword">val</span> sample1 = Sample(</span><br><span class="line">    field1 = <span class="string">&quot;foo&quot;</span>,</span><br><span class="line">    field2 = <span class="literal">true</span>,</span><br><span class="line">    field3 = <span class="number">1</span>,</span><br><span class="line">    field4 = <span class="number">1f</span>,</span><br><span class="line">    field5 = <span class="number">1.0</span></span><br><span class="line">)</span><br><span class="line"><span class="keyword">val</span> sample2 = Sample(field2=<span class="literal">true</span>)</span><br><span class="line"><span class="keyword">val</span> sample3 = Sample()</span><br></pre></td></tr></table></figure><br>To duplicate a new instance, you just need to specify the changing part.<br><figure class="highlight kotlin"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">val</span> sample4 = sample3.copy(field2=<span class="literal">true</span>, field3=<span class="number">2</span>)</span><br></pre></td></tr></table></figure></p>
<h4 id="Delegation-Item-16-Favor-composition-over-inheritance"><a href="#Delegation-Item-16-Favor-composition-over-inheritance" class="headerlink" title="Delegation (Item 16: Favor composition over inheritance)"></a>Delegation (Item 16: Favor composition over inheritance)</h4><p>It is common that an OOP beginner treats inheritance as a way to reuse code. As the example in the book says, if we need to create an <code>InstrumentedHashSet</code>, one may do something like this:<br><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">public</span> <span class="keyword">class</span> <span class="title class_">InstrumentedHashSet</span>&lt;E&gt; <span class="keyword">extends</span> <span class="title class_">HashSet</span>&lt;E&gt; &#123;</span><br><span class="line">    <span class="comment">// The number of attempted element insertions</span></span><br><span class="line">    <span class="keyword">private</span> <span class="type">int</span> <span class="variable">addCount</span> <span class="operator">=</span> <span class="number">0</span>;</span><br><span class="line">    <span class="keyword">public</span> <span class="title function_">InstrumentedHashSet</span><span class="params">()</span> &#123; &#125;</span><br><span class="line"></span><br><span class="line">    <span class="meta">@Override</span> <span class="keyword">public</span> <span class="type">boolean</span> <span class="title function_">add</span><span class="params">(E e)</span> &#123;</span><br><span class="line">        addCount++;</span><br><span class="line">        <span class="keyword">return</span> <span class="built_in">super</span>.add(e);</span><br><span class="line">    &#125;</span><br><span class="line">    <span class="meta">@Override</span> <span class="keyword">public</span> <span class="type">boolean</span> <span class="title function_">addAll</span><span class="params">(Collection&lt;? extends E&gt; c)</span> &#123;</span><br><span class="line">        addCount += c.size();</span><br><span class="line">        <span class="keyword">return</span> <span class="built_in">super</span>.addAll(c);</span><br><span class="line">    &#125;</span><br><span class="line">    <span class="keyword">public</span> <span class="type">int</span> <span class="title function_">getAddCount</span><span class="params">()</span> &#123;</span><br><span class="line">        <span class="keyword">return</span> addCount;</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br>This is totally reasonable because we don&#39;t want to repeat ourselves. By extending <code>HashSet</code>, we can get the full functionality of the <code>Set</code> interface, and we only need to override the method which we want to customize. However, the code doesn&#39;t work, even if it does work, we still should not do this, because the <code>HashSet</code> is not designed for inheritance, you should read the book if you fail to understand this, we won&#39;t dive deep here. Anyway, the book suggests we use composition and forwarding instead, That is:<br><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// Wrapper class - uses composition in place of inheritance</span></span><br><span class="line"><span class="keyword">public</span> <span class="keyword">class</span> <span class="title class_">InstrumentedSet</span>&lt;E&gt; <span class="keyword">extends</span> <span class="title class_">ForwardingSet</span>&lt;E&gt; &#123;</span><br><span class="line">    <span class="keyword">private</span> <span class="type">int</span> <span class="variable">addCount</span> <span class="operator">=</span> <span class="number">0</span>;</span><br><span class="line">    <span class="keyword">public</span> <span class="title function_">InstrumentedSet</span><span class="params">(Set&lt;E&gt; s)</span> &#123;</span><br><span class="line">        <span class="built_in">super</span>(s);</span><br><span class="line">    &#125;</span><br><span class="line">    <span class="meta">@Override</span> <span class="keyword">public</span> <span class="type">boolean</span> <span class="title function_">add</span><span class="params">(E e)</span> &#123;</span><br><span class="line">        addCount++;</span><br><span class="line">        <span class="keyword">return</span> <span class="built_in">super</span>.add(e);</span><br><span class="line">    &#125;</span><br><span class="line">    <span class="meta">@Override</span> <span class="keyword">public</span> <span class="type">boolean</span> <span class="title function_">addAll</span><span class="params">(Collection&lt;? extends E&gt; c)</span> &#123;</span><br><span class="line">        addCount += c.size();</span><br><span class="line">        <span class="keyword">return</span> <span class="built_in">super</span>.addAll(c);</span><br><span class="line">    &#125;</span><br><span class="line">    <span class="keyword">public</span> <span class="type">int</span> <span class="title function_">getAddCount</span><span class="params">()</span> &#123;</span><br><span class="line">        <span class="keyword">return</span> addCount;</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br><span class="line"><span class="comment">// Reusable forwarding class</span></span><br><span class="line"><span class="keyword">public</span> <span class="keyword">class</span> <span class="title class_">ForwardingSet</span>&lt;E&gt; <span class="keyword">implements</span> <span class="title class_">Set</span>&lt;E&gt; &#123;</span><br><span class="line">    <span class="keyword">private</span> <span class="keyword">final</span> Set&lt;E&gt; s;</span><br><span class="line">    <span class="keyword">public</span> <span class="title function_">ForwardingSet</span><span class="params">(Set&lt;E&gt; s)</span> &#123; <span class="built_in">this</span>.s = s; &#125;</span><br><span class="line">    <span class="keyword">public</span> <span class="keyword">void</span> <span class="title function_">clear</span><span class="params">()</span> &#123; s.clear();&#125;</span><br><span class="line">    <span class="keyword">public</span> <span class="type">boolean</span> <span class="title function_">contains</span><span class="params">(Object o)</span> &#123; <span class="keyword">return</span> s.contains(o);&#125;</span><br><span class="line">    <span class="keyword">public</span> <span class="type">boolean</span> <span class="title function_">isEmpty</span><span class="params">()</span> &#123; <span class="keyword">return</span> s.isEmpty();&#125;</span><br><span class="line">    <span class="keyword">public</span> <span class="type">int</span> <span class="title function_">size</span><span class="params">()</span> &#123; <span class="keyword">return</span> s.size();&#125;</span><br><span class="line">    <span class="keyword">public</span> Iterator&lt;E&gt; <span class="title function_">iterator</span><span class="params">()</span> &#123; <span class="keyword">return</span> s.iterator();&#125;</span><br><span class="line">    <span class="keyword">public</span> <span class="type">boolean</span> <span class="title function_">add</span><span class="params">(E e)</span> &#123; <span class="keyword">return</span> s.add(e);&#125;</span><br><span class="line">    <span class="keyword">public</span> <span class="type">boolean</span> <span class="title function_">remove</span><span class="params">(Object o)</span> &#123; <span class="keyword">return</span> s.remove(o);&#125;</span><br><span class="line">    <span class="keyword">public</span> <span class="type">boolean</span> <span class="title function_">containsAll</span><span class="params">(Collection&lt;?&gt; c)</span> &#123; <span class="keyword">return</span> s.containsAll(c);&#125;</span><br><span class="line">    <span class="keyword">public</span> <span class="type">boolean</span> <span class="title function_">addAll</span><span class="params">(Collection&lt;? extends E&gt; c)</span> &#123; <span class="keyword">return</span> s.addAll(c);&#125;</span><br><span class="line">    <span class="keyword">public</span> <span class="type">boolean</span> <span class="title function_">removeAll</span><span class="params">(Collection&lt;?&gt; c)</span> &#123; <span class="keyword">return</span> s.removeAll(c);&#125;</span><br><span class="line">    <span class="keyword">public</span> <span class="type">boolean</span> <span class="title function_">retainAll</span><span class="params">(Collection&lt;?&gt; c)</span> &#123; <span class="keyword">return</span> s.retainAll(c);&#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br>As you can see, The <code>ForwordingSet</code> does nothing but forward every method call to an existing Set implementation. As the book says: &quot;It&#39;s tedious to write forwarding methods, but you have to write the forwarding class for each interface only once.&quot; However, even mediocre programmers like me don&#39;t like write such code, it just like the meaningless getter,setter in Java, even worse, the IDE may not be able to generate such code for you.<br>In Kotlin, it is done by delegation, the equivalent code is<br><figure class="highlight kotlin"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">class</span> <span class="title class_">InstrumentSet</span>&lt;<span class="type">T</span>&gt;(</span><br><span class="line">    <span class="keyword">private</span> <span class="keyword">val</span> <span class="keyword">set</span>: MutableSet&lt;T&gt;</span><br><span class="line">) : MutableSet&lt;T&gt; <span class="keyword">by</span> <span class="keyword">set</span> &#123;</span><br><span class="line"></span><br><span class="line">    <span class="keyword">var</span> addCount = <span class="number">0</span></span><br><span class="line">        <span class="keyword">private</span> <span class="keyword">set</span></span><br><span class="line"></span><br><span class="line">    <span class="keyword">override</span> <span class="function"><span class="keyword">fun</span> <span class="title">add</span><span class="params">(element: <span class="type">T</span>)</span></span>: <span class="built_in">Boolean</span> &#123;</span><br><span class="line">        addCount++</span><br><span class="line">        <span class="keyword">return</span> <span class="keyword">set</span>.add(element)</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    <span class="keyword">override</span> <span class="function"><span class="keyword">fun</span> <span class="title">addAll</span><span class="params">(elements: <span class="type">Collection</span>&lt;<span class="type">T</span>&gt;)</span></span>: <span class="built_in">Boolean</span> &#123;</span><br><span class="line">        addCount += elements.size</span><br><span class="line">        <span class="keyword">return</span> <span class="keyword">set</span>.addAll(elements)</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br>The <code>by</code> keyword indicates we are forwarding every <code>Set</code> method to the set property.</p>
<h4 id="Variance-Item-28-Use-bounded-wildcards-to-increase-API-flexibility"><a href="#Variance-Item-28-Use-bounded-wildcards-to-increase-API-flexibility" class="headerlink" title="Variance (Item 28: Use bounded wildcards to increase API flexibility)"></a>Variance (Item 28: Use bounded wildcards to increase API flexibility)</h4><p>Suppose we are writing a log method, In some cases, we may want to reduce the runtime cost by taking a Supplier as a parameter.<br><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">void</span> <span class="title function_">log</span><span class="params">(Supplier&lt;Object&gt; supplier)</span> &#123;</span><br><span class="line">    <span class="keyword">if</span> (isLogEnabled) &#123;</span><br><span class="line">        System.out.println(supplier.get());</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br>With this API, If get the logging message is expensive, we can call <code>debug(() -&gt; expensiveToStringOperation())</code> instead of <code>debug(expensiveToStringOperation())</code>(the latter one need to evaluate the &quot;expensiveToStringOperation&quot; even when <code>isLogEnabled</code> is false, this is as known as call by value). Suppose somehow we have already defined the Supplier&#39;s type as <code>Supplier&lt;String&gt;</code>, we cannot pass it in since <code>Supplier&lt;String&gt;</code> is not a subtype of <code>Supplier&lt;Object&gt;</code>. Although this as a totally safe operation because if the method is able to handle any Object, It must also be able to handle a String.<br><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">Supplier&lt;String&gt; supplier = () -&gt; <span class="string">&quot;hello world&quot;</span>;</span><br><span class="line"><span class="comment">// doesn&#x27;t compile</span></span><br><span class="line">Console.log(supplier);</span><br><span class="line"><span class="comment">// doesn&#x27;t compile either</span></span><br><span class="line">Console.log((Supplier&lt;Object&gt;) supplier);</span><br></pre></td></tr></table></figure><br>To increase the flexibility of our log method, as the Effective Java recommended, we should rewrite our log method as<br><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// Supplier&lt;? extends Object&gt; can be simplified to Supplier&lt;?&gt;</span></span><br><span class="line"><span class="comment">// But I leave it here to explain the bound</span></span><br><span class="line"><span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">void</span> <span class="title function_">log</span><span class="params">(Supplier&lt;? extends Object&gt; supplier)</span> &#123;</span><br><span class="line">    <span class="keyword">if</span> (isLogEnabled) &#123;</span><br><span class="line">        System.out.println(supplier.get());</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br>Such that we can pass the <code>Supplier&lt;String&gt;</code> in. However, write these wildcard types correctly might be tricky, the book also introduces a mnemonic to help us determine which wildcard type to use, which says:</p>
<blockquote>
<p>PECS stands for producer-extends, consumer-super</p>
</blockquote>
<p>That is, if we only use the parameter as a producer, we should use the <code>&lt;? extends T&gt;</code> form, else if we only use the parameter as a consumer, we use the <code>&lt;? super T&gt;</code> form. In the previous example, we only use the supplier as a producer, so we use extends bound.<br>However, in this case, the <code>Supplier</code> can<br>But the point is, how could you ever use Supplier as a Consumer? It is not possible. In other words, a <code>Supplier&lt;String&gt;</code> should always be a subtype of <code>Supplier&lt;Object&gt;</code>, no matter how do you use it. Similarly, a <code>Consumer&lt;Object&gt;</code> should always be a subtype of <code>Consumer&lt;String&gt;</code>, this is so called covariance and contravariance.<br>In Kotlin, if your class only &quot;produce&quot; or &quot;consume&quot; a type parameter. The compiler will help you decide which kind of &quot;variance&quot; your class is allowed. Therefore, you don&#39;t need to write the wildcard everywhere, the <code>Supplier&lt;String&gt;</code> will automatically become subtype of <code>Supplier&lt;Object&gt;</code>, the following code compiles correctly.<br><figure class="highlight kotlin"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">val</span> supplier: Supplier&lt;String&gt; = Supplier &#123;</span><br><span class="line">    <span class="string">&quot;hello world&quot;</span></span><br><span class="line">&#125;</span><br><span class="line">Console.debug(supplier)</span><br></pre></td></tr></table></figure><br>In Kotlin, The bounded wildcard parameter is only needed when your parameter class can be used as both consumer <strong>and</strong> producer, and your method only uses it as consumer <strong>or</strong> producer. Although, it is extremely rare to encounter such a situation, A possibly but not practical signature would be<br><figure class="highlight kotlin"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">fun</span> <span class="type">&lt;T&gt;</span> <span class="title">copyData</span><span class="params">(src: <span class="type">Deque</span>&lt;<span class="type">out</span> <span class="type">T</span>&gt;, dest: <span class="type">Deque</span>&lt;<span class="type">in</span> <span class="type">T</span>&gt;)</span></span> &#123;</span><br><span class="line">    dest.addAll(src)</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br>I guess this time, the mnemonic should be updated to:</p>
<blockquote>
<p>POCI stands for producer-out, consumer-in</p>
</blockquote>
<h3 id="Conclusion"><a href="#Conclusion" class="headerlink" title="Conclusion"></a>Conclusion</h3><p>Java is dear, Kotlin is dearer.</p>
]]></content>
      
        <categories>
            
            <category> Programming Language </category>
            
        </categories>
        
        
        <tags>
            
            <tag> Kotlin </tag>
            
            <tag> Java </tag>
            
        </tags>
        
    </entry>
    
    <entry>
      <title><![CDATA[When "Soft Delete" Meets "Unique Index"]]></title>
      <url>https://blog.staynoob.cn/post/2019/05/when-soft-delete-meets-unique-index/</url>
      <content type="html"><![CDATA[<blockquote>
<p>Do some casually writing to practice my English.</p>
</blockquote>
<p>Recently, I was asked to enable soft delete for all the tables I created, it sounds like a breeze, as an experienced noob, I &quot;finished&quot; it immediately without even think about it. This is how I did, add a boolean column named &quot;deleted&quot; for each table, then replace every unique index to include the &quot;deleted&quot; column, done! Anyway, It turns out I was too naive.</p>
<h3 id="What-39-s-wrong-with-my-naive-solution"><a href="#What-39-s-wrong-with-my-naive-solution" class="headerlink" title="What&#39;s wrong with my naive solution?"></a>What&#39;s wrong with my naive solution?</h3><p>Imagine that we have a user table:<br><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">CREATE</span> <span class="keyword">TABLE</span> `<span class="keyword">user</span>` (</span><br><span class="line">  `id` <span class="type">int</span>(<span class="number">11</span>) <span class="keyword">NOT</span> <span class="keyword">NULL</span> AUTO_INCREMENT,</span><br><span class="line">  `username` <span class="type">varchar</span>(<span class="number">50</span>) <span class="keyword">NOT</span> <span class="keyword">NULL</span>,</span><br><span class="line">  `deleted` tinyint(<span class="number">1</span>) <span class="keyword">NOT</span> <span class="keyword">NULL</span> <span class="keyword">DEFAULT</span> <span class="number">0</span>,</span><br><span class="line">  <span class="keyword">PRIMARY</span> KEY (`id`),</span><br><span class="line">  <span class="keyword">UNIQUE</span> KEY `uq_user` (`username`,`deleted`)</span><br><span class="line">)</span><br></pre></td></tr></table></figure><br>Whenever we need to &quot;soft delete&quot; a user, we set the value of the &quot;deleted&quot; column to 1, what could possibly go wrong?<br>Now, let&#39;s say we have a user &quot;John Snow&quot;, we deleted the corresponding record with the following command after he was killed in GOT season 5.<br><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">update</span> <span class="keyword">user</span> <span class="keyword">set</span> deleted <span class="operator">=</span> <span class="number">1</span> <span class="keyword">where</span> id <span class="operator">=</span> #&#123;id&#125;;</span><br></pre></td></tr></table></figure><br>then we insert it again after they bring him back in season 6.<br><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">insert</span> <span class="keyword">user</span>(username) <span class="keyword">values</span> (&quot;John Snow&quot;);</span><br></pre></td></tr></table></figure><br>Everything works smoothly so far, except that we won&#39;t able to delete him again. this time <code>update user set deleted = 1 where id = #&#123;id&#125;;</code> will raise a duplicate records error, that is exactly what the unique constraint does, but apparently, it violates our intention.<br>The problem is, we only want the username to be unique if the user is active, we don&#39;t care if there are multiple deleted user share a username. In other words, we only want a partially unique constraint which restricts the active user.</p>
<span id="more"></span>
<h3 id="Partially-Index-in-PostgreSQL"><a href="#Partially-Index-in-PostgreSQL" class="headerlink" title="Partially Index in PostgreSQL"></a>Partially Index in PostgreSQL</h3><p>If you are using PostgreSQL, you are lucky, they do have implemented the &quot;Partially index&quot;. the create statement may look like:<br><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">CREATE</span> <span class="keyword">TABLE</span> &quot;user&quot; (</span><br><span class="line">    id <span class="type">int</span> GENERATED <span class="keyword">BY</span> <span class="keyword">DEFAULT</span> <span class="keyword">AS</span> <span class="keyword">IDENTITY</span> <span class="keyword">PRIMARY</span> KEY,</span><br><span class="line">    username <span class="type">varchar</span>(<span class="number">50</span>) <span class="keyword">NOT</span> <span class="keyword">NULL</span>,</span><br><span class="line">    deleted <span class="type">boolean</span> <span class="keyword">NOT</span> <span class="keyword">NULL</span> <span class="keyword">DEFAULT</span> <span class="literal">false</span></span><br><span class="line">);</span><br><span class="line"><span class="keyword">CREATE</span> <span class="keyword">UNIQUE</span> INDEX uq_user <span class="keyword">ON</span> &quot;user&quot; <span class="keyword">USING</span> btree(username) <span class="keyword">WHERE</span> <span class="keyword">NOT</span> deleted;</span><br></pre></td></tr></table></figure><br>Now everything works as our expectations.</p>
<h3 id="In-absense-of-partially-index"><a href="#In-absense-of-partially-index" class="headerlink" title="In absense of partially index"></a>In absense of partially index</h3><p>However, the majority database products don&#39;t have the &quot;Partially index&quot; concept. We have to take a workaround, instead of storing the deleted flag as a boolean value, now we have to use a &quot;delete token&quot;. This is how it is done, if a record is active, we keep the token to be 0, if we want to delete a record, we set the &quot;delete token&quot; column to a unique value to avoid violating unique constraint. A perfect delete token could be the timestamp when you want to delete the record(the auto-generated id of the deleted record is also a fair choice). The following script shows how to do it with MySQL.<br><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">CREATE</span> <span class="keyword">TABLE</span> `<span class="keyword">user</span>` (</span><br><span class="line">  `id` <span class="type">int</span>(<span class="number">11</span>) <span class="keyword">NOT</span> <span class="keyword">NULL</span> AUTO_INCREMENT,</span><br><span class="line">  `username` <span class="type">varchar</span>(<span class="number">50</span>) <span class="keyword">NOT</span> <span class="keyword">NULL</span>,</span><br><span class="line">  `deleted_at` <span class="type">int</span>(<span class="number">11</span>) unsigned <span class="keyword">NOT</span> <span class="keyword">NULL</span> <span class="keyword">DEFAULT</span> <span class="number">0</span>,</span><br><span class="line">  <span class="keyword">PRIMARY</span> KEY (`id`),</span><br><span class="line">  <span class="keyword">UNIQUE</span> KEY `uq_user` (`username`,`deleted_at`)</span><br><span class="line">)</span><br></pre></td></tr></table></figure><br>Note that <code>deleted_at timestamp DEFAULT NULL</code> doesn&#39;t work since most databases do not treat multiple null as duplicate values, which means multiple active user with a same username can pass the unique constraint.<br>Also, the statement of deleting a user has now becoming<br><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">UPDATE</span> <span class="keyword">user</span> <span class="keyword">SET</span> deleted_at <span class="operator">=</span> unix_timestamp() <span class="keyword">WHERE</span> id <span class="operator">=</span> #&#123;id&#125;;</span><br></pre></td></tr></table></figure><br>Now, we are finally able to &quot;kill&quot; John Snow multiple times.</p>
]]></content>
      
        <categories>
            
            <category> Backend </category>
            
        </categories>
        
        
        <tags>
            
            <tag> Database </tag>
            
        </tags>
        
    </entry>
    
    <entry>
      <title><![CDATA[The Good Old Transaction]]></title>
      <url>https://blog.staynoob.cn/post/2019/05/the-good-old-transaction/</url>
      <content type="html"><![CDATA[<blockquote>
<p>随便写写跟事务相关的笔记</p>
</blockquote>
<h2 id="ACID"><a href="#ACID" class="headerlink" title="ACID"></a>ACID</h2><h3 id="原子性-Atomicity"><a href="#原子性-Atomicity" class="headerlink" title="原子性(Atomicity)"></a>原子性(Atomicity)</h3><p>这里的原子性含义与多线程编程中的原子性有一些细微的区别，在多线程语境中，如果一个方法满足原子性，则其它线程无法看到该方法执行的中间状态，但它并不保证该方法中的语句全生效或全不生效（All or Nothing）。相反，ACID 中的原子性保证 All or Nothing，但其并不保证其它事务是否能看到该事务执行的中间状态，在 ACID 中，该属性由隔离性(Isolation)来保证。考虑下面这段程序<br><figure class="highlight kotlin"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">var</span> counter: <span class="built_in">Int</span> = <span class="number">0</span>;</span><br><span class="line"></span><br><span class="line"><span class="meta">@Synchronized</span></span><br><span class="line"><span class="function"><span class="keyword">fun</span> <span class="title">increase</span><span class="params">()</span></span> &#123;</span><br><span class="line">    counter++;</span><br><span class="line">    <span class="keyword">if</span> (ThreadLocalRandom.current().nextBoolean())</span><br><span class="line">        <span class="keyword">throw</span> Exception(<span class="string">&quot;Oops!&quot;</span>)</span><br><span class="line">    counter++;</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="meta">@Synchronized</span></span><br><span class="line"><span class="function"><span class="keyword">fun</span> <span class="title">printCurrent</span><span class="params">()</span></span> &#123;</span><br><span class="line">    println(counter);</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br>这里 @Synchronized 保证 increase 方法是符合原子性的，这意味着，如果没有异常出现，则 printCurrent 方法不可能打印出一个奇数。但如果出现异常，counter 的第一次自增并不会回滚，也就是说这次 increase 调用只将 counter 自增1。与其相对的是下面这段 SQL 代码：<br><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">begin</span> transaction;</span><br><span class="line"><span class="keyword">update</span> counter <span class="keyword">set</span> <span class="keyword">value</span> <span class="operator">=</span> <span class="keyword">value</span> <span class="operator">+</span> <span class="number">1</span> <span class="keyword">where</span> id <span class="operator">=</span> <span class="number">1</span>;</span><br><span class="line">if ROUND(RAND(),<span class="number">0</span>)<span class="operator">=</span><span class="number">1</span></span><br><span class="line"><span class="keyword">begin</span>;</span><br><span class="line">    THROW <span class="number">50000</span>, <span class="string">&#x27;Oops!&#x27;</span>, <span class="number">1</span></span><br><span class="line"><span class="keyword">end</span>;</span><br><span class="line"><span class="keyword">update</span> counter <span class="keyword">set</span> <span class="keyword">value</span> <span class="operator">=</span> <span class="keyword">value</span> <span class="operator">+</span> <span class="number">1</span> <span class="keyword">where</span> id <span class="operator">=</span> <span class="number">1</span>;</span><br><span class="line"><span class="keyword">commit</span>;</span><br></pre></td></tr></table></figure><br>即便没有异常出现，如果没有 Isolation（或者 Isolation.level = READ_UNCOMMMITTED），则其它事务能看到这段代码的中间状态，但如果有异常出现，第一次自增的操作会被回滚。<br>从这个角度来说，ACID 中的 Atomicity 更多的指的是在错误出现时能够自动撤销之前修改，也许把 &quot;A&quot; 理解成 Abortability 更恰当。</p>
<h3 id="一致性-Consistency"><a href="#一致性-Consistency" class="headerlink" title="一致性(Consistency)"></a>一致性(Consistency)</h3><p>ACID 中的一致性，表示事务只会将数据从一种“正确”的状态修改成另一种“正确”的状态。举例来说如果说有一个用户交易系统，所有的事务只会把金额从一个账户转移到另一个账户，那么可以保证的是无论执行多少次转账交易，该系统所有账户的余额都是“正确”的。<br>这里的“正确”之所以要打引号是因为它是由应用定义的，除了一些外键约束，唯一约束之外，数据库并不能理解当前的数据是否符合你对“正确”的定义。<br>换句话说，原子性，隔离性，持久性是数据库的属性，但一致性可能更应该被看成应用的属性，应用通过数据库提供的原子性，隔离性来保证数据的一致性。因此 &quot;C&quot; 并不真的属于 “ACID”（It was said that the C in ACID was &quot;tossed in to make the acronym work&quot;）。</p>
<span id="more"></span>
<h3 id="隔离性-Isolation"><a href="#隔离性-Isolation" class="headerlink" title="隔离性(Isolation)"></a>隔离性(Isolation)</h3><p>隔离性意味着并发执行的事务应该相互隔离，换句话说，即便在现实中会有大量事务并发执行，数据库系统应该保证这些并发事务执行结果跟按时间顺序一个接一个执行的结果是一样的，很容易想到这是一个需要牺牲性能才能满足的属性，因此大部分数据库产品会提供一些较“弱”的隔离级别供用户选择。</p>
<h3 id="持久性-Durability"><a href="#持久性-Durability" class="headerlink" title="持久性(Durability)"></a>持久性(Durability)</h3><p>持久性意味着一旦事务执行成功，其带来的修改不会丢失，单从字面上看，这其实是一个无法实际达成的属性，因此这里可以将其理解成，一旦事务执行成功，意味着其修改已被写入某种永久存储介质（比如硬盘），类似于进程被终止或停电等问题不会造成数据丢失。</p>
<h2 id="反常现象与隔离级别-Anomalies-and-Isolation-Level"><a href="#反常现象与隔离级别-Anomalies-and-Isolation-Level" class="headerlink" title="反常现象与隔离级别(Anomalies and Isolation Level)"></a>反常现象与隔离级别(Anomalies and Isolation Level)</h2><p>在现实中，并发与竞争条件往往处理起来非常复杂，且容易出错。因此数据库产品希望通过 Isolation 将这部分复杂的逻辑转移至存储引擎（就像 JDK 要实现线程安全的集合一样），然而正如前文所提到的那样，实现完整的 Isolation 对数据库的性能影响过大，所以数据库一般会提供一些“弱”隔离级别(Isolation Level)，并注明在这些隔离级别下，会出现哪些可能的反常现象（不幸的是，这些反常现象，隔离级别以及对应的实现在数据库领域并没有很好的标准化。也就是说不同的隔离级别在不同的数据库产品中可能有不同的叫法，它们能够防止的异常现象也可能不一样，下文的内容只能够作为参考）。</p>
<h3 id="脏写-Dirty-Write"><a href="#脏写-Dirty-Write" class="headerlink" title="脏写(Dirty Write)"></a>脏写(Dirty Write)</h3><p>假设我们有一个在线买车的应用，买车操作涉及两个步骤，首先在 listing 表中更新指定车的买家，之后在 invoices 表中更新发票的接收者，下面的例子中，Alice 与 Bob 同时购买同一辆车。<br><img src="/img/content/the-good-old-transaction/7-5.jpeg" alt="dirty-write"><br>由于 Alice 更新 listing 表的操作被 Bob 覆盖，Bob 更新 invoices 表的操作被 Alice 覆盖，最终导致的结果是 Bob 是车的买家，但 Alice 成了发票的接收人。这种现象被称为脏写（注意区分脏写与下文提到的丢失更新现象，脏写是覆盖未提交的修改，丢失更新是覆盖已提交的修改）。<br>大部分数据库通过行级锁(row-level lock)来防止脏写，如果一个事务想要修改一条记录，首先需要获取锁，并一直持有锁直到事务最终提交或回滚，如果其它事务想要修改同一行，则修改必须阻塞直到它能够获取到锁为止（注意这种方式并不能防止丢失更新）。</p>
<h3 id="脏读-Dirty-Read"><a href="#脏读-Dirty-Read" class="headerlink" title="脏读(Dirty Read)"></a>脏读(Dirty Read)</h3><p>如果一个事务可以读取到其它事务未提交的修改（即其它事务执行的中间状态），也就违反了上面多线程语境中的 Atomicity（更糟糕的是，由于事务的 Abortability，你可能会读到其它事务已经回滚的修改），这种现象被称为脏读。<br>脏读也可以通过同样的行级锁机制来避免，即要求事务在读取行数据时先获取锁，读取完再释放，这样可以保证如果某行数据被某个事务修改，在该事务释放锁之前其它事务都无法读取到该行数据（IBM DB2使用这种方式）。但大部分数据库产品不这么做，而是在行数据被修改时，维持该行数据修改前后的两个副本。当没有锁的事务读取时，返回修改前的版本，当持有锁的事务读取时，返回修改后的版本。</p>
<h3 id="不可重复读-Nonrepeatable-Read-Read-Skew"><a href="#不可重复读-Nonrepeatable-Read-Read-Skew" class="headerlink" title="不可重复读(Nonrepeatable Read/Read Skew)"></a>不可重复读(Nonrepeatable Read/Read Skew)</h3><p>假设我们有一个银行系统，Alice 有两个账户，分别存了500块钱，Alice 试图将100快钱从账户2转移至账户1，并在转账的同时分别查询两个账户的余额，它可能会看到如下情况<br><img src="/img/content/the-good-old-transaction/7-6.jpeg" alt="nonrepeatable-read"><br>Alice 的查询事务先查询账户1，发现余额是500元，再查询账户2，发现余额是400元，资金总和成了900元，这会让人产生迷惑（这里如果它最后在同一事务中再次查询账户1，会得到余额600元）。同一个事务中，两次读取同一条数据返回结果却不一样。这种现象称为不可重复读（或读偏移）。<br>大部分数据库产品采用快照隔离来实现可重复读，其核心原则是读取不阻塞写入，写入也不阻塞读取，它的实现思路与之前通过保持数据修改前，修改后的版本来防止脏读类似。不同的是之前只需要维护行数据的两个版本（即已提交的版本与已修改但未提交的版本），但实现快照隔离则需要维护行数据的多个版本，这种方法又称为多版本并发控制（multiversion concurrency control）。<br>一般来说如果一个存储引擎实现了快照隔离，它也可以复用这些快照来实现提交读隔离级别，如果当前事务是提交读隔离级别，则每次查询同一行数据有可能返回不同的快照，但如果是可重复读隔离级别，则每次查询只返回同一快照。<br>下图演示了 PostgreSQL 中实现快照隔离的大致方案，当一个事务开始时，为其分配一个全局单调递增的事务id，如果该事务对数据做出了修改，被修改的数据将同时保存该事务id：<br><img src="/img/content/the-good-old-transaction/7-7.jpeg" alt="snapshot-isolation"><br>表中的每条行数据都有 create_by 与 delete_by 两个字段，分别记录了插入事务与删除事务的事务id，具体的实现方式如下：</p>
<ul>
<li>新增<br>在 create_by 字段记录创建者的事务id。</li>
<li>删除<br>在 delete_by 字段记录删除者的事务id。之后当垃圾回收机制确认没有任何事务能够访问到该行数据时，再将其真正清除。</li>
<li>修改<br>将修改转化成删除与创建，举例来说，事务13从账户2将账户2的余额从500改成了400，现在账户2在账户表中存在两条记录，一条余额为500，被标记为 deleted by tx13，另一条余额为400，被标记为 created by tx13。</li>
<li><p>查询<br>查询按如下方式工作：</p>
<ol>
<li>在事务开始时，数据库为其生成一个列表，记录其它正在进行中的事务，忽略由这些事务做出的任何修改。</li>
<li>忽略任何 aborted 的事务的修改（即忽略所有回滚的事务修改）。</li>
<li>忽略任何事务id大于当前事务所做出的修改。</li>
<li>剩下的其它修改对查询可见。</li>
</ol>
<p>换句话说，数据仅当满足如下条件才可见：</p>
<ul>
<li>在查询事务开始时，创建该数据的事务已提交。</li>
<li>数据没有被标记成删除，或已被标记为删除，但在查询事务开始时，将其标记为删除的事务还没有提交。</li>
</ul>
</li>
</ul>
<p>快照隔离是很常用的隔离级别，但在不同的数据库可能有不同的叫法，Oracle 中称其为 serializable，PostgreSQL 和 MySQL 称其为 repeatable read。在 IBM DB2 中使用 repeatable read 来指代 serializability，最终导致没有人知道 repeatable read 到底指的是哪种隔离级别。</p>
<h3 id="丢失更新-Lost-Updates"><a href="#丢失更新-Lost-Updates" class="headerlink" title="丢失更新(Lost Updates)"></a>丢失更新(Lost Updates)</h3><p><img src="/img/content/the-good-old-transaction/7-1.jpeg" alt="lost-updates"><br>即当两个事务并发的执行读取-修改-写入时，其中一个事务做出的修改并另一个事务覆盖。</p>
<ol>
<li>原子操作<br> 如果读取-修改-写入符合原子性（注意这里指的是并发编程中的 Atomicity 而不是 ACID 中的 &quot;Abortability&quot;），那么其它事务无法看到该流程的中间状态，也就是说不会发生丢失更新。大部分数据库保证如下 SQL 语句是符合原子性的 <figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">update</span> counters <span class="keyword">set</span> <span class="keyword">value</span> <span class="operator">=</span> <span class="keyword">value</span> <span class="operator">+</span> <span class="number">1</span> <span class="keyword">where</span> key <span class="operator">=</span> <span class="string">&#x27;foo&#x27;</span>;</span><br></pre></td></tr></table></figure>
 但是这种方式有个非常明显的缺点，如果你需要先读取数据，执行一些检查逻辑，最后再决定要不要 increase，这种方法就不适用了。</li>
<li>显式排它锁（悲观锁）<br> 之前讨论如何防止脏写现象时提到过行级锁，在事务更新行数据时需要先获取锁，之后持有该锁直至事务最终提交或回滚。这种方式只能防止脏写，无法防止丢失更新，因为它只阻塞其它事务的写操作，不阻塞读操作。因此我们可以显式使用 for update 语句来为行数据添加排它锁，该锁会阻塞其它事务的读操作<strong>（ 注意，其它事务必须也使用 select ... for update 语句读，换句话说，select for update 阻塞 update 与其它 select for update，但不阻塞 select ）</strong>，从而达到防止丢失更新的目的。 <figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">begin</span> transaction;</span><br><span class="line"><span class="keyword">select</span> <span class="keyword">value</span> <span class="keyword">from</span> counters <span class="keyword">where</span> key <span class="operator">=</span> <span class="string">&#x27;foo&#x27;</span> <span class="keyword">for</span> <span class="keyword">update</span>;</span><br><span class="line"><span class="comment">-- some logic</span></span><br><span class="line"><span class="keyword">update</span> counters <span class="keyword">set</span> <span class="keyword">value</span> <span class="operator">=</span> <span class="keyword">value</span> <span class="operator">+</span> <span class="number">1</span> <span class="keyword">where</span> key <span class="operator">=</span> <span class="string">&#x27;foo&#x27;</span>;</span><br><span class="line"><span class="keyword">commit</span>;</span><br></pre></td></tr></table></figure></li>
<li>Compare-and-set（乐观锁）<br> CAS 是并发编程中一种非常常见的用于防止丢失更新的方式，大致的方案见如下代码 <figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">begin</span> transaction;</span><br><span class="line"><span class="keyword">select</span> <span class="keyword">value</span> <span class="keyword">from</span> counters <span class="keyword">where</span> key <span class="operator">=</span> <span class="string">&#x27;foo&#x27;</span>;</span><br><span class="line"><span class="comment">-- some logic</span></span><br><span class="line"><span class="keyword">update</span> counters <span class="keyword">set</span> <span class="keyword">value</span> <span class="operator">=</span> <span class="keyword">value</span> <span class="operator">+</span> <span class="number">1</span> <span class="keyword">where</span> key <span class="operator">=</span> <span class="string">&#x27;foo&#x27;</span> <span class="keyword">and</span> <span class="keyword">value</span> <span class="operator">=</span> oldvalue</span><br><span class="line"><span class="keyword">commit</span>;</span><br></pre></td></tr></table></figure>
 注意上面代码中的 where 条件增加了 <code>value=oldvalue</code>，如果有其它用户在该事务执行中途修改了 counter 值，则该事务将执行失败，对比悲观锁使用阻塞，乐观锁使用的失败+重试来达成防止丢失更新的目的。这里值得注意的一点是，考虑到某些数据库产品对快照隔离的具体实现，where 从句可能会读到事务开始时的快照，也就意味着 <code>value=oldvalue</code> 可能恒成立，这种方法就失去效果了。</li>
<li>自动探测丢失更新<br> 如果数据库实现了快照隔离，丢失更新现象很容易被自动探测到。PostgreSQL 的 repeatable read, Oracle 的 serailizable 和 SQLSever 的 snapshot isolation 都会自动探测丢失更新并终止事务，需要注意的是 <strong>MySQL 的 repeatable read 隔离级别不会探测丢失更新</strong>，因此有人认为 MySQL 并没有真正提供快照隔离。</li>
</ol>
<h3 id="幻读-Phantom-Read-Write-Skew"><a href="#幻读-Phantom-Read-Write-Skew" class="headerlink" title="幻读(Phantom Read/Write Skew)"></a>幻读(Phantom Read/Write Skew)</h3><p>假设我们实现了一个医院的排班系统，医生允许通过该应用请假，但需要保证在任意时刻至少有一名医生在值班。我们实现请假的思路是先检查当前正在值班的人数，只有在人数大于等于2的前提下才允许医生请假。现在假设 Alice 和 Bob 正在值班，它们都想请假，并几乎同时在应用中按下了请假按钮，下图展示了可能发生的事：<br><img src="/img/content/the-good-old-transaction/7-8.jpeg" alt="write-skew"><br>由于 Alice 与 Bob 几乎在同时请求当班人数，该请求都返回2，于是两个人同时通过了该约束，进入应用代码的下一阶段，最后结果是两个人都请假成功，导致无人值班，这种现象称为写偏移(write skew)。下面是另外两个常见的例子：</p>
<ul>
<li>房间/机票预定系统<br>  假设我们希望保证一个房间不被多个用户同时预定，可能的流程如下  <figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">begin</span> transaction;</span><br><span class="line"><span class="keyword">select</span> <span class="built_in">count</span>(<span class="operator">*</span>) <span class="keyword">from</span> bookings <span class="keyword">where</span> room_id<span class="operator">=</span><span class="number">123</span></span><br><span class="line"><span class="keyword">and</span> end_time <span class="operator">&gt;</span> t1 <span class="keyword">and</span> start_time <span class="operator">&lt;</span> t2;</span><br><span class="line"><span class="comment">-- check if the query return zero</span></span><br><span class="line"><span class="keyword">insert</span> <span class="keyword">into</span> bookings <span class="keyword">values</span> (<span class="number">123</span>, t1, t2, user_id);</span><br><span class="line"><span class="keyword">commit</span>;</span><br></pre></td></tr></table></figure>
  同样，多个事务并发执行时，快照隔离无法保证不会有多个用户在同一时段预定该房间。</li>
<li>注册用户<br>  在用户注册时，我们可能希望用户名是唯一的，如果你在应用层做唯一检查，那么快照隔离级别无法保证最终用户名的唯一性。（好在大部分数据库提供唯一约束来实现该目的）</li>
</ul>
<p>上面所有的例子基本上都有着类似的模式：</p>
<ol>
<li>用户先执行 select 语句，然后判断执行结果是否满足一些特定条件（比如说查询某时段值班人数，某时段房间的预定记录，某用户名是否存在）。</li>
<li>基于第一次查询结果来执行一些操作。</li>
<li>第二步执行的操作会影响到第一步的查询结果。</li>
</ol>
<p>进一步可以发现，这些现象发生的根本原因是因为在一个事务中所做出的修改，可能改变另一个事务中某条 select 语句的查询结果，这种现象又被称为幻读（同一个事务中，多次执行同样的查询可能得到不同的结果集）。</p>
<h4 id="冲突具体化-Materializing-Conflicts"><a href="#冲突具体化-Materializing-Conflicts" class="headerlink" title="冲突具体化(Materializing Conflicts)"></a>冲突具体化(Materializing Conflicts)</h4><p>在排班系统中，第三步针对第一步返回的结果集做 update 操作，因此我们可以通过在第一步中使用 <code>select for update</code> 排它锁来避免写偏移。但在其它例子中，我们的判定结果依赖于第一步返回空集合，即第三步针对第一步返回的结果集做 insert 操作，在这种场景下，我们没有行记录可以加锁，为了找到合适于加锁的行对象，我们可以让冲突具体化，举例来说，在预定房间的例子中，我们可以为每个可能的房间 + 预定时段组合生成一条记录，这样我们就可以通过 <code>select for update</code> 语句甚至于乐观锁来保证不会发生冲突(如果你觉得生成这样的记录太麻烦，也可以降级到只锁预定时段或房间记录，锁的粒度越大则系统的效率越低，你甚至可以只锁一条 isBooking 的记录来实现同步，本质上来说这就跟用数据库实现分布式锁没什么区别了)。<br>这么做的缺点是需要应用代码的介入，而且细粒度的控制可能会非常麻烦（比如说你需要定时预生成针对每个房间未来一段时间的预定记录），因此在性能允许的情况下，更推荐使用串行化隔离级别来解决这个问题（注意这里并不是让你将数据库默认的隔离级别设置成串行化，而是只针对特定事务将其设为串行化）。</p>
<h4 id="串行化-Serializability"><a href="#串行化-Serializability" class="headerlink" title="串行化(Serializability)"></a>串行化(Serializability)</h4><p>串行化被认为是最“严格”的隔离级别，它保证即使事务是并发执行的，但其执行结果跟按时间顺序一个接一个执行的结果是一样的。换句话说，如果你使用该隔离级别，那么前面所有由并发，竞争条件所带来的问题都不需要考虑了。下面简单介绍一些串行化的实现方法。</p>
<h5 id="按顺序执行-Actual-Serial-Execution"><a href="#按顺序执行-Actual-Serial-Execution" class="headerlink" title="按顺序执行(Actual Serial Execution)"></a>按顺序执行(Actual Serial Execution)</h5><p>最直观的方法就是真的在存储引擎中使用单线程按顺序执行事务，尽管这种方法很明显，但人们直到 2007 年左右才开始意识到这是一种可行的方法，导致大家重新重视这种方法的原因主要有两个：</p>
<ul>
<li>随着内存降价，将整个数据集放入内存中已经称为可行的方案，如果整个数据集都在内存中，那么事务的执行会比以前快很多。</li>
<li>数据库设计者开始意识到 OLTP 事务一般持续时间不会太长，也不会做太多的读取和写入。而 OLAP 事务一般是只读的，因此它们可以在多线程环境下使用快照隔离级别执行。</li>
</ul>
<p>这种方法的核心依赖在于每个事务必须能够快速执行，在传统的事务中，应用服务器与数据库服务器需要产生多次交互（发送查询命令，接收查询结果等），一个事务执行大部分时间都花在网络 IO 中。如果让数据库服务器单线程执行这样的事务，其性能可想而知。因此，如果要单线程串行执行事务，一般会让应用服务器以存储过程的形式将整个事务逻辑一次性发送至数据库服务器，这样一来就不用花时间等待网络与磁盘 IO 了。但依赖于存储过程也就成了这种方法最大的缺点，毕竟应该没有程序员喜欢写存储过程吧，其次，如果一个事务需要涉及到多个数据分区，这种方法的性能也会受到严重影响。</p>
<blockquote>
<p>注：目前使用顺序执行的数据库有 Redis, Datomic。</p>
</blockquote>
<h5 id="二阶段锁-Two-Phase-Locking"><a href="#二阶段锁-Two-Phase-Locking" class="headerlink" title="二阶段锁(Two-Phase Locking)"></a>二阶段锁(Two-Phase Locking)</h5><p>二阶段锁类似于之前用于防止脏写现象的行级锁，只不过比之前更加严格：</p>
<ul>
<li>如果事务 A 已经读取了行数据，事务 B 想要修改这行数据，则事务 B 需要阻塞直到事务 A 提交或终止。</li>
<li>如果事务 A 修改了一行数据，事务 B 想要读取该行数据，则事务 B 需要阻塞直到事务 A 提交或终止（而不是读取一个 A 的快照）。</li>
</ul>
<p>之前我们提到过快照隔离的核心原则是读取不阻塞写入，写入也不阻塞读取。相比之下，二阶段锁则更像并发编程语境中的写锁（排它锁）与读锁（共享锁），每个行锁都有排它与共享两种模式：</p>
<ol>
<li>如果一个事务想要读取一条记录，必须先获取该条记录的读锁。</li>
<li>如果一个事务想要修改一条记录，必须先获取该条记录的写锁。</li>
<li>如果一个事务先读取，再修改一条记录，则需要将它的读锁升级成写锁。</li>
<li>如果一个事务获取了锁，则必须持有锁直至事务提交或终止（让人有点迷惑的是，二阶段锁的名字出自这里，第一阶段是事务运行阶段，第二阶段是事务终止阶段）。</li>
</ol>
<p>（注意：由于大量使用锁机制，因此该实现很容易出现死锁，数据库必须能够自动检测死锁并终止其中一个造成死锁的事务）<br>细心的同学可能会发现，二阶段锁依然没有解决之前由于幻读造成的写倾斜问题。为此我们还需要引入<strong>断言锁(Predicate Locks)</strong>的概念，即我们不仅需要对行对象加锁，还需要对未来可能出现的行对象加锁，拿之前预定房间的例子来说：<br><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">select</span> <span class="operator">*</span> <span class="keyword">from</span> bookings </span><br><span class="line"><span class="keyword">where</span> room_id<span class="operator">=</span><span class="number">123</span> <span class="keyword">and</span> end_time <span class="operator">&gt;</span> t1 <span class="keyword">and</span> start_time <span class="operator">&lt;</span> t2;</span><br></pre></td></tr></table></figure><br>我们需要对上面 where 从句匹配的整个区域加锁，即便该区域目前可能没有记录。<br>断言锁与行级锁的工作方式类似，即</p>
<ul>
<li>如果事务 A 想要读取匹配指定条件的所有记录，需要先获取该条件的读锁。</li>
<li>如果事务 A 想要插入，修改，删除一条记录，需要先检查修改前与修改后的记录是否跟匹配某个条件锁，如果是的话，事务 A 需要获取该条件的写锁。</li>
</ul>
<p>可以看到这种方法需要对大量的写操作做条件匹配运算，因此并不实用，大部分数据库采用的是一种近似的实现方法，也就是<strong>索引区域锁(Index-range Locks)</strong>，以上面的查询条件 <code>where room_id=123 and end_time &gt; t1 and start_time &lt; t2</code> 来说，该查询涉及到三个索引，分别是 room_id, start_time, end_time：</p>
<ul>
<li>假设数据库使用 room_id 作为第一索引来执行查询，可以直接在 room_id=123 索引上加共享锁，该锁预示有一个事务正在查询房间123的所有预定记录。</li>
<li>假设数据库先使用 start_time 和 end_time 索引执行查询，则可以直接锁一个索引区域，预示有一个事务在查询该时间段的预定记录。</li>
</ul>
<p>换句话说，索引区域锁通过锁定一个大于等于断言锁的区域来避免多次进行条件匹配运算，如果实在没有合适的索引可以用来加锁，则数据库可以直接对整张表加锁。</p>
<blockquote>
<p>注：目前使用二阶段锁的数据库有 MySQL(InnoDB), SQLServer。</p>
</blockquote>
<h5 id="串行快照隔离-Serializable-Snapshot-Isolation-SSI"><a href="#串行快照隔离-Serializable-Snapshot-Isolation-SSI" class="headerlink" title="串行快照隔离(Serializable Snapshot Isolation(SSI))"></a>串行快照隔离(Serializable Snapshot Isolation(SSI))</h5><p>之前在讨论如何防止丢失更新时我们已经讨论过悲观锁与乐观锁，悲观并发控制的思路是，如果两个操作并发执行有可能会出错，那么假设它一定会出错，因此需要阻塞其中一个操作。相反，乐观并发控制的思路是假设并发执行一定不会出错，在事务最终提交前，再通过某种机制来检查是否有错。换句话说，悲观并发控制更倾向于阻塞，乐观并发控制倾向于失败与重试。从防止幻读这个角度来说，按顺序执行与二阶段锁都是采用悲观并发控制的思路，而串行快照隔离则采用乐观并发控制思路，下面简单介绍它的实现思路。<br>从之幻读的例子中，我们可以总结出一种模式，即一个事务先查询某些记录，然后依据该查询结果决定后续的操作，在快照隔离级别下，等到事务提交时，这个查询结果可能已经被其它事务修改了。因此，一旦数据库探测到某个事务的某次查询结果已经被其它事务修改了，则该事务后续的所有写请求可能都是不安全的，数据库必须阻止该事务提交。剩下的问题在于如何探测这些“过期”的查询（stale reads）。下面简单介绍快照隔离下，可能出现过期查询的情况以及串行快照的处理方式：</p>
<ul>
<li>在读取之前，存在其它事务未提交的修改，依据快照隔离的实现，这种情况查询的结果可能是已过期的快照。<br>  为了防止这种情况发生，我们需要去跟踪并收集那些依据 MVCC 可见规则而忽略掉的修改的事务 id，最后在事务提交时，检查这些事务 id 中是否有成功提交的事务，如果有则终止当前事务。</li>
<li>在读取之后，其它事务修改数据并提交。<br>  在之前介绍二阶段锁时我们提到过索引锁，即在事务读取某个索引区域的时候，先获取该索引区域的读锁，并阻塞其它试图修改该索引段的事务。这里我们可以采取类似的机制，但我们只记录哪些事务读过该索引区域，当有事务试图修改该索引区域时，并不阻塞该事务，而是通知之前读取过该索引段的事务它们可能读到了过期数据。</li>
</ul>
<p>相比其它两种实现，这种方法最大的优势在于读取不会阻塞写入，最大的缺点在于，一个事务涉及的语句越多，执行的时间越长，就越有可能遇到冲突而回滚，换句话说，在高并发环境下，一个耗时较长的事务可能一直无法提交成功。</p>
<blockquote>
<p>注：目前使用 SSI 的数据库有 PostgreSQL(since version 9.1), FoundationDB。</p>
</blockquote>
<h2 id="轻量级事务-Light-weighted-Transaction"><a href="#轻量级事务-Light-weighted-Transaction" class="headerlink" title="轻量级事务(Light-weighted Transaction)"></a>轻量级事务(Light-weighted Transaction)</h2><p>前文可以看到，总的来说越严格的隔离级别意味着越低的性能，如果还要考虑数据分区（partition)，这个现象会更加明显。因此更为年轻的 &quot;NoSQL&quot; 数据库几乎都不支持传统的事务。相反，它们声称自己支持“轻量级”事务，以 Cassandra 为例：<br><figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">-- insert</span></span><br><span class="line"><span class="keyword">insert</span> <span class="keyword">into</span> <span class="keyword">user</span> (id, name) <span class="keyword">values</span> (<span class="string">&#x27;123&#x27;</span>, <span class="string">&#x27;foo&#x27;</span>) if <span class="keyword">not</span> <span class="keyword">exists</span>;</span><br><span class="line"><span class="comment">-- update</span></span><br><span class="line"><span class="keyword">update</span> <span class="keyword">user</span> <span class="keyword">set</span> name <span class="operator">=</span> <span class="string">&#x27;bar&#x27;</span> <span class="keyword">where</span> id <span class="operator">=</span> <span class="string">&#x27;123&#x27;</span> if name <span class="operator">=</span> <span class="string">&#x27;foo&#x27;</span>;</span><br></pre></td></tr></table></figure><br>上面是两条 CQL 语句展示了两个轻量级事务（注意 if 从句），可以看到这跟之前我们用 where 从句实现的乐观锁没什么不同。说白了就是提供了一个 CAS 原语，因此我更倾向于认为所谓的“轻量级事务”只是一个市场营销用语。</p>
<blockquote>
<p>注：轻量级事务(Light-weighted Transaction)又被称为单行事务(Single-Object Transaction)，与其对应的传统事务又被称为多行事务(Multi-Object Transaction)</p>
</blockquote>
<p>在没有多行事务支持的情况下，业界出现了大量的 workaround 方案，其中比较出名的有 <a href="https://queue.acm.org/detail.cfm?id=1394128">BASE</a>，<a href="https://dzone.com/articles/transactions-for-the-rest-of-us">TCC</a>，<a href="https://www.enterpriseintegrationpatterns.com/ramblings/18_starbucks.html">Compensate　Transaction</a>，我不否认它们的价值，但个人认为，这些方法的本质都是在业务上做出妥协，不同的业务需求可能有不同的妥协方式，它们最多只能被视为完成交易的方案，跟数据库领域的事务并没有什么关系，偏偏某些发布这些方案的人还试图用其特定业务场景的正确性来证明自己实现了”分布式事务“（要知道即便是 2PC 也只是一个原子提交协议，仅仅实现了事务的 &quot;Abortability&quot; 属性而已）。<br>好在随着技术的发展，近几年新兴的一些数据库产品（Google Spanner, TiDB）据说确实能够实现真正意义的分布式事务，尽管我非常好奇它们是如何做到的，无奈最近已经开始工作了，短期内应该是没有时间去研究了。</p>
<h2 id="Reference"><a href="#Reference" class="headerlink" title="Reference"></a>Reference</h2><ul>
<li><a href="https://item.jd.com/12186665.html">Designing Data-Intensive Applications</a></li>
</ul>
]]></content>
      
        <categories>
            
            <category> Backend </category>
            
        </categories>
        
        
        <tags>
            
            <tag> Database </tag>
            
        </tags>
        
    </entry>
    
    <entry>
      <title><![CDATA[分布式锁真的“安全”吗？]]></title>
      <url>https://blog.staynoob.cn/post/2019/03/is-distributed-lock-safe/</url>
      <content type="html"><![CDATA[<blockquote>
<p>今天偶然间读到了 Martin Kleppmann 与 Salvatore Sanfilippo 关于 Redlock 算法是否”安全“的讨论，觉得挺有启发的，因此打算把目前的思考记下来。由于这篇文章比较长，这里提前剧透我的结论，“所有带有效期的分布式锁本质上都是不“安全”的，只有“安全”的资源服务，没有“安全”的分布式锁”。</p>
</blockquote>
<h3 id="背景"><a href="#背景" class="headerlink" title="背景"></a>背景</h3><p>Martin Kleppmann 是剑桥大学分布式系统领域的一名研究员，同时也是 <a href="https://item.jd.com/12186665.html">Designing Data-Intensive Applications</a> 这本书的作者，他在个人博客中发了一篇文章 <a href="https://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html">How to do distributed locking</a>，其中涉及了大量对 Redlock 算法安全性的质疑，Salvatore Sanfilippo（Redis 的创始人，也是这里 Redlock 算法的作者）随后发表 <a href="http://antirez.com/news/101">Is Redlock safe?</a> 回应这些质疑，这篇文章总结了这两篇文章讨论的重点和我对这些问题的想法。</p>
<h3 id="术语和约定"><a href="#术语和约定" class="headerlink" title="术语和约定"></a>术语和约定</h3><p>像之前的翻译文章一样，一些专业术语翻译成中文反而不好理解，这里提前解释一下这些术语。</p>
<ul>
<li>safety 属性<br>  简单说 safety 就是保证不会有坏事发生。如果该属性被违背，我们一般可以确切的知道它们在哪个时间点被违背，比如说集合元素的唯一性就是 safety 属性。如果一个集合插入了一个重复元素，那么在插入的这个时间点违反了唯一性这个 safety 属性。（注意不要混淆这里的 safety 属性和文章标题中“安全”一词的含义）</li>
<li>liveness 属性<br>  简单说，liveness 就是保证好事最终会发生。比如说最终一致性就是 liveness 属性（一般 liveness 属性定义中都包含”最终“二字）</li>
</ul>
<blockquote>
<p>&quot;Intuitively, a safety property describes what is allowed to happen, and a liveness property describes what must happen.&quot;</p>
</blockquote>
<p>为了更好的描述问题，我们先定义下面三种角色：</p>
<ul>
<li>资源服务：即需要被锁保护的资源。</li>
<li>锁服务：即本文 Redlock 算法扮演的角色。</li>
<li>锁用户：申请与释放锁的客户端。（下文可能简称为用户）</li>
</ul>
<p>使用分布式锁的目的主要有两种，分别是：</p>
<ol>
<li>效率(Efficiency)：通过锁来避免多次做重复的工作，计算重复的内容等等。这种场景下即便偶然出现多个用户同时持有锁，并同时与资源服务发生交互，也是可以忍受的。</li>
<li>正确性(Correctness)：也就是文章标题所说的“安全”，我们希望资源服务在锁的保护下能够做“正确”的事。更严谨的说，我们希望任一时刻，只有一个用户能够访问资源服务，而且即便锁在该用户在与资源服务交互的中途过期，也不至于破坏资源服务的一致性。</li>
</ol>
<p>无论出于哪种目的，单从分布式锁服务的角度来说，我们都希望它具有如下属性（下文将以属性1，属性2，属性3来引用这些属性）：</p>
<ol>
<li>互斥（safety 属性）：在任一时刻，只有一个用户能持有锁。</li>
<li>避免死锁（liveness 属性）：每把锁都有一个有效期，超出有效期则自动释放锁。如果没有这样的自动释放机制，那么一个已获得锁的用户宕机或失联，将导致资源被持续锁定直至该用户故障被修复，在大部分场景中，这是不可接受的。</li>
<li>容错（liveness 属性）：没有单点失败问题，只要系统中多数锁服务节点正常工作，用户就能够获取和释放锁。</li>
</ol>
<p>下文讨论的 RedLock 算法期望解决的主要问题是单点 Redis 作为分布式锁服务时无法满足属性3，下面先来了解一下该算法。</p>
<span id="more"></span>
<h3 id="Redlock-算法"><a href="#Redlock-算法" class="headerlink" title="Redlock 算法"></a>Redlock 算法</h3><p>Redlock 算法的实现基于单点分布式锁，下面是单个 Redis 实例实现分布式锁的方式。<br>首先是用户获取锁的命令<br><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">SET key value NX PX 30000</span><br></pre></td></tr></table></figure><br>这里 key 可能是某个你想锁的资源名，value 是某个全局唯一的随机值（该值后面会用于释放锁），这条命令在 key 不存在的前提下(NX选项)，设置 key 对应的 value 值，30000 毫秒后，该 key 会过期（PX选项）。该命令执行成功则代表成功获取锁，否则代表已经有其它用户先获取了锁。下面是释放锁的代码<br><figure class="highlight lua"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">if</span> redis.call(<span class="string">&quot;get&quot;</span>,KEYS[<span class="number">1</span>]) == ARGV[<span class="number">1</span>] <span class="keyword">then</span></span><br><span class="line">    <span class="keyword">return</span> redis.call(<span class="string">&quot;del&quot;</span>,KEYS[<span class="number">1</span>])</span><br><span class="line"><span class="keyword">else</span></span><br><span class="line">    <span class="keyword">return</span> <span class="number">0</span></span><br><span class="line"><span class="keyword">end</span></span><br></pre></td></tr></table></figure><br>如果你跟我一样看不懂 lua 脚本也没关系，这段代码的目的是比较 key(resource_name) 对应的 value 值是否是否等于之前设定的随机值，如果等于就删除该 key 来释放锁（注意该脚本的执行是符合原子性的）。之所以这么做是为了防止用户释放其它用户持有的锁（一个用户可能并不知道锁已经因为过期而被其它用户持有）。<br>这样单节点 Redis 服务器已经满足了属性1与属性2，但是该节点宕机将导致整个服务不可用，因此我们需要想办法进一步满足属性3。一个比较 naive 的做法是添加 redis 服务器做主从切换(failover)，即 master 服务器不可用时自动将 slave 服务器晋升为 master。但这么做无法满足属性1，因为 Redis 主从数据复制是异步的，考虑下面的执行序列：</p>
<ol>
<li>用户A在 master 服务器获取到锁。</li>
<li>在数据从 master 复制到 slave 之前，master 宕机。</li>
<li>slave 成为新的 master。</li>
<li>用户B在新的 master 服务器获取到同一把锁，最终用户A与用户B持有相同的锁（违背属性1）。</li>
</ol>
<p>下面来看看 Redlock 算法，假设我们有 N 个相互独立的 Redis 节点（这里先假设 N=5），用户按照下列操作来获取与释放锁：</p>
<ol>
<li>获取当前时间戳</li>
<li>使用相同的 key, value 依次在所有节点上“获取锁”（方式跟之前在单节点上获取锁是一样的），每个节点获取锁的时间控制在一个比较小的范围内。举例来说，如果锁的有效时间是 10 秒，那么在每个节点获取锁的时间最好控制在 5-50 毫秒左右，这样才能保证如果某个 Redis 节点宕机或失联，整个获取锁操作不会阻塞太久。换句话说，如果单个锁服务实例不可用，尽快尝试在下一个实例获取锁。</li>
<li>用户计算步骤2所花费的时间（使用当前时间减去第一步保存的时间戳），当且仅当用户在多数锁服务节点（此例中3个节点）中成功获取到锁，并且获取锁花费的时间小于锁的过期时间，才算成功获取到锁。</li>
<li>如果步骤3获取锁成功，锁的实际有效时间是初始有效时间减去步骤2所花费的时间，比如锁的有效时间设定为 10 秒，步骤2花掉了 1 秒，那么锁的实际有效时间是 9 秒。</li>
<li>如果步骤3获取锁失败，尝试在所有节点上执行解锁操作（方式跟之前单节点上解锁是一样的）</li>
</ol>
<p>以上基本就是 Redlock 算法的全部内容，相比单点 Redis 来说，它的作用是提供更高的可用性，即满足属性3。</p>
<h3 id="争议点"><a href="#争议点" class="headerlink" title="争议点"></a>争议点</h3><p>在理解 Martin Kleppmann 与 Salvatore Sanfilippo 讨论的话题之前，需要先理解一些概念，这里我暂且把它们称作争议点。</p>
<h4 id="1-Fencing-Token"><a href="#1-Fencing-Token" class="headerlink" title="1. Fencing Token"></a>1. Fencing Token</h4><p>下图展示了一个失败的分布式锁的执行序列<br><img src="/img/content/is-distributed-lock-safe/unsafe-lock.png" alt="unsafe-lock"><br>图中用户1在获取锁后进程暂停（导致进程暂停的原因可能有很多种，详情可以阅读 Kleppmann 的书或者该博客原文），在进程暂停期间，锁超时自动释放，于是用户2获得了锁，之后用户1从暂停中恢复。最终结果是用户1与用户2都认为自己拥有锁，资源服务也会同时受理用户1与用户2的请求，因此违背了属性1。<br>Kleppmann 建议使用一个全局单调递增的 &quot;Fencing token&quot; 来解决这样的问题。如图所示：<br><img src="/img/content/is-distributed-lock-safe/fencing-tokens.png" alt="fencing-tokens"><br>在每次用户获得锁时，锁服务同时为该用户分配一个全局单调递增的 token，我们要求每次用户请求资源服务时带上该 token。资源服务一旦受理过 token 值更高的请求，就拒绝其它 token 较低的请求。上图中用户1首先获得一个 token 为33的锁，之后进程暂停，锁超时自动释放，用户2获得 token 为34的锁。最后资源服务只受理用户2的请求。如果你使用 Zookeeper 作为分布式锁服务，它的 zxid 或者 znode version 都可以用来作为 &quot;Fencing token&quot;.</p>
<h4 id="2-Wall-Clock-与-Monotonic-Clock"><a href="#2-Wall-Clock-与-Monotonic-Clock" class="headerlink" title="2. Wall Clock 与 Monotonic Clock"></a>2. Wall Clock 与 Monotonic Clock</h4><p>现代计算机一般至少会提供两种时钟，分别是 Wall Clock 和 Monotonic Clock。</p>
<ul>
<li>Wall Clock(又称 Time of day clock, Real Time)<br>  即 UNIX 系统中调用 <code>clock_gettime(CLOCK_REALTIME)</code>，JVM 中调用 <code>System.currentTimeMillies()</code> 得到的时间。该时间一般需要通过网络与 NTP(Network Time Protocol) 服务器同步，因此可能会突然跳到未来的某个时间点，或者跳回过去。而且它也可能被系统管理员手动设置，再加上还有闰秒问题。这些因素导致该时钟不适用于测量耗时。</li>
<li>Monotonic Clock<br>  即 UNIX 系统中调用 <code>clock_gettime(CLOCK_MONOTONIC)</code>，JVM 中调用 <code>System.nanoTime()</code> 得到的时间。正如它名字一样，该时间不会回退，同时它也不受 NTP 影响，因此非常适合测量时间区间。但单个 Monotonic Clock 时间值没有任何意义，比较不同电脑中的 Monotonic Clock 时间值也没意义。</li>
</ul>
<h4 id="3-Asynchronous-System-Model"><a href="#3-Asynchronous-System-Model" class="headerlink" title="3. Asynchronous System Model"></a>3. Asynchronous System Model</h4><p>在学术界，对于分布式算法来说，最理想的系统模型是 <a href="http://courses.csail.mit.edu/6.852/08/papers/CT96-JACM.pdf">asynchronous model with unreliable failure detectors</a>，该模型对时间不做任何假设，即进程可能暂停任意时间，网络中的包可能延迟任意久，时钟可能会任意跳跃。这些问题一般不会影响该模型下算法的 safety 属性，只有 liveness 属性才依赖于过期时间(timeout)或其它失败检测手段(failure detector)。换言之，就是在出现进程暂停，网络延迟，时钟跳跃等情况时，算法的性能可能毫无保障，但它至少不会做错误的决策。</p>
<h3 id="Kleppmann-的质疑"><a href="#Kleppmann-的质疑" class="headerlink" title="Kleppmann 的质疑"></a>Kleppmann 的质疑</h3><ul>
<li>关于 Fencing Token<br>Redlock 算法中并不存在任何机制用于生成全局单调递增的 token，它为锁提供的唯一随机数并不保证单调性。而简单使用一个 Redis 节点作为计数器又存在单点问题。在分布式系统中，你可能需要自己采用某种共识算法来生成这样的 token。</li>
<li>关于 Clock<br>Redis 使用 Wall Clock 而不是 Monotonic Clock 来判断 key 是否过期，这很容易导致 key 的过期比想象中快很多，或者慢很多。</li>
<li><p>关于算法对时间的依赖<br>Redlock 的 safety 属性过多的依赖于对时间的假设（Using time to solve consensus）。它假设所有的 Redis 节点持有 key 的时间约等于分布式锁的过期时间；假设网络延迟相比锁的有效期来说要小很多；假设进程暂停时间比锁的有效期小得多。<br>因此，如果时钟跳跃，Redlock 算法将无法满足属性1，假设锁服务由 A, B, C, D, E 五个 Redis 节点组成，考虑下面的执行序列（下文将以“示例1”来引用这个例子）：</p>
<ol>
<li>用户1在节点 A, B, C 中获得锁，由于网络原因，D, E 节点不可达。</li>
<li>C 节点时钟跳跃，锁在 C 节点中过期。</li>
<li>用户2在节点 C, D, E 中获得锁，由于网络原因，A, B 节点不可达。</li>
<li>用户1与用户2持有同一把锁（违背属性1）。</li>
</ol>
<p>这个问题不止在时钟跳跃时会发生，如果节点C的 Redis 进程在将数据持久化到磁盘前被杀掉，然后立即重启（key 数据丢失），也有可能会发生同样的情况，因此 Redlock 文档中建议延迟节点的重启时间，使其至少与最长的锁有效期一样长。但是这种延迟重启策略再次依赖于节点C能够精准测量时间。<br>Kleppmann 在原文中还额外补充了另一个由进程暂停和网络延迟导致违背 safety 属性的例子（下文将以“示例2”来引用这个例子）。</p>
<ol>
<li>用户1在节点 A, B, C, D, E 请求锁。</li>
<li>当请求成功的响应还未返回到用户1时，用户1进程暂停。</li>
<li>锁过了有效期（A, B, C, D, E节点各自删除对应 key）。</li>
<li>用户2在节点 A, B, C, D, E 获得锁。</li>
<li>用户1从暂停中恢复，收到成功获得锁的信息。</li>
<li>用户1与用户2持有同一把锁（违背属性1）。</li>
</ol>
</li>
</ul>
<h3 id="Sanfilippo-的回应"><a href="#Sanfilippo-的回应" class="headerlink" title="Sanfilippo 的回应"></a>Sanfilippo 的回应</h3><ul>
<li>关于 Fencing Token<br>这个问题并不是 Redlock 独有的，还有很多带自动释放机制的分布式锁服务中都不提供单调递增的计数器。主要原因是：<ol>
<li>实现所谓的 &quot;Fencing Token&quot; 机制需要资源服务的积极配合。在分布式锁的大量使用场景里，我们没办法控制资源服务。如果我们有办法控制资源服务如何处理用户的请求，或许我们也不需要分布式锁了。</li>
<li>即便一定要 &quot;Fencing Token&quot;，它也没必要是单调递增的，任何全局唯一 id 都可以用作 token。比如 Redlock 算法中每次申请锁时用的那个随机 value 值就可以用作 token。每次用户获取锁成功时，先设置资源服务的 currentToken，后续的每次请求都带上该 token，资源服务受理用户请求时如果发现 token 不一致就拒绝请求。简单说就是把 <code>if (request.token &lt; currentToken)</code> 改成 <code>if (request.token != currentToken)</code>。同样可以保证同一时刻，只有一个用户允许访问资源服务。</li>
</ol>
</li>
<li>关于 Clock（Redis 使用 Wall-Clock）<br>Sanfilippo 承认这是 Redis 的缺陷，并表示后续会修复这个问题。</li>
<li><p>关于算法对时间的依赖<br>根据 Kleppmann 的批评，Redlock 对时间的依赖主要包含三个部分：</p>
<ol>
<li>时钟依赖：所有的 Redis 节点持有 key 的时间需要约等于分布式锁的过期时间。</li>
<li>网络延迟依赖：网络延迟不能太高（相比锁的有效期来说）</li>
<li>进程暂停时间依赖：进程暂停时间不能太长（相比锁的有效期来说）</li>
</ol>
<p><strong>关于时钟依赖</strong>：Redlock 的 safety 属性并不依赖于各 Redis 节点的时钟是否有误差，而是依赖于各节点之间时间流逝的速度是否近似相等。比方说，各节点能将计时 5s 的误差控制在 10% 以内就行。影响这个误差的两个主要原因是：</p>
<ol>
<li>系统管理员手动调整时钟</li>
<li>ntpd 守护进程修改时钟（也就是前文说的 Wall Clock 同步网络时间的机制）</li>
</ol>
<p>这两个问题都是可以避免的，问题1很简单，让管理员别这么做就行了。如果要考虑人为干预的话那管理员还可以 <code>echo foo &gt; /my/raft/log.bin</code> 来破坏 Raft 算法。问题2可以通过改用平滑修改时间的 ntpd 来避免。再说如果改用 Monotonic Clock。这两个问题也就都不存在了。<br><strong>关于网络延迟依赖</strong>：Redlock 算法在步骤三会计算获取锁用掉的时间，如果有包括网络延迟在内的任何原因导致获取锁用掉的时间多于锁的有效时间，则获取锁失败。所以网络延迟不会影响算法的 safety 属性。<br><strong>关于进程暂停时间依赖</strong>：进程暂停如果发生在算法的步骤三之前，那它造成的影响等同于网络延迟，最终只可能导致获取锁失败。如果发生在步骤三之后（即用户认为自己获取锁成功之后），那结果等同于争议点一(Fencing Token)的执行序列。</p>
</li>
</ul>
<h3 id="我对分歧的看法"><a href="#我对分歧的看法" class="headerlink" title="我对分歧的看法"></a>我对分歧的看法</h3><p>（个人认为）两篇文章表达的分歧有：</p>
<ul>
<li>分歧1：Kleppmann 认为安全的分布式锁服务需要同时提供一个全局递增的 token。Sanfilippo 由于资源服务常常不可控，因此该 token 没必要</li>
<li>分歧2：即使 &quot;Fencing Token&quot; 有必要，出于 &quot;Fencing&quot; 的目的，全局唯一与全局递增也没什么区别。</li>
<li>分歧3：Kleppmann 认为 Redlock 的安全性过多的依赖于对时间的假设（参见示例1与示例2）。Sanfilippo 并不赞同，因为示例2中网络延迟与进程暂停如果发生在算法的步骤三之前，则问题并不存在，如果发生在步骤三之后，则等价于 Fencing Token 问题（这里我没看到关于示例1的解释，但个人觉得示例1依然等价于 Fencing Token 问题，原因后面会提到）。</li>
</ul>
<p>这些分歧其实都出于对属性1的描述（在任意时刻，只有一个用户能持有锁）不够严谨。假设有 A, B 两个用户，针对同一资源，我们理想中的属性1应该进一步拆分成如下属性：</p>
<ul>
<li>属性a: 在任意时刻，锁服务都知道自己是否已授权锁，如果已授权，锁服务知道具体授权给了哪个用户（这里假设授权给了用户A）。</li>
<li>属性b: 在任意时刻，最多只有一个用户认为自己获得了锁。</li>
<li>属性c: 在任意时刻，资源服务最多只允许一个用户访问。</li>
<li>属性d: 在任意时刻，如果锁已授权给一个用户，资源服务只允许当前锁服务授权的那个用户访问（用户A）。</li>
</ul>
<p>首先我们可以确定的是，在异步系统模型，锁会自动释放的前提下，可以得到：</p>
<blockquote>
<p>结论1：属性b不可能满足。<br>证明：在不对时间做任何假设的情况下，用户以为自己持有锁，但锁已经过期，这种情况是无法避免的，参见&quot;Fencing Token&quot;中的第一个例子与示例2。</p>
</blockquote>
<p>现在不妨先来看分歧3，包括 Sanfilippo 自己也认为 Redlock 的正确性依赖于所有 Redis 节点中，时间以接近的速度流逝。但我认为并非如此，因为如果现实并不满足这条假设，那么算法只是无法满足属性b（注意这里无所谓算法是否可以满足属性b，因为反正结论1告诉我们属性b不可能满足），但该算法依然保证了属性a。在示例1中，c节点发生时钟跳跃，最终导致的结果其实等价于锁提前过期，在时钟不可靠的情况下，锁提前或延后过期总会发生，这对属性a而言并没有影响。换言之即使使用单节点 Redis 锁，该节点时钟同样可能发生跳跃，锁同样可能提前过期。因此<strong>示例1的问题同样等价于 &quot;Fencing Token&quot; 中的问题，它们本质上都是锁用户并不知道自己持有的锁已经过期的问题</strong>。即便你的锁不设有效期，而是使用 Zookeeper 临时节点，通过 Zookeeper session（对用户做心跳验证） 来判断锁是否过期，该问题依旧存在。<br>同时，基于结论1，我们可以得到：</p>
<blockquote>
<p>结论2：如果资源服务不可控，属性c也不可能满足。<br>证明：由于属性b无法满足，势必会出现某个时刻，两个用户同时认为自己拥有锁，而此时如果资源服务不受控（即无法使用Fencing Token），则属性c无法满足。</p>
</blockquote>
<p>这样一来分歧1可以用一句话概括，<strong>是否需要 &quot;Fencing Token&quot; 取决于你的系统是否需要满足属性c</strong>，换句话说，如果你的目的是 Efficiency，则 Redlock 算法和其它分布式锁机制一样，是“安全”的。如果你的目的是 Correctness，则 Redlock 是不“安全”的。<br>最后是分歧2，出于 &quot;fencing&quot; 的目的，全局唯一的 token 是否等价于全局递增的 token？假设我们照 Sanfilippo 所言在资源服务中使用全局唯一的 token 验证用户是否真的持有锁，考虑下面的执行序列：</p>
<ol>
<li>用户A获得锁后，进程暂停，锁过期。</li>
<li>用户B获得锁后，将资源服务的 currentToken 设成 B.token。</li>
<li>用户A从暂停中恢复，将资源服务的当前 currentToken 设成 A.token。</li>
<li>锁服务认为 B 持有锁，但资源服务认为 A 持有锁（满足属性c，不满足属性d）。</li>
</ol>
<p>这样看来，使用全局递增的 token 就能解决这个问题，但再考虑下面的执行序列：</p>
<ol>
<li>用户A获得锁后，进程暂停，锁过期。</li>
<li>用户B获得锁后，进程暂停。</li>
<li>用户A从暂停中恢复，继续访问资源服务。</li>
<li>锁服务认为 B 持有锁，但直到用户 B 与资源服务通话之前，资源服务都认为 A 持有锁（满足属性c，不满足属性d）。</li>
</ol>
<p>这里的结论是，全局唯一与全局递增的 token 都只能满足属性c，无法满足属性d，但如果使用递增 token，资源服务能够在收到 B 的请求后，第一时间意识到 B 是较近的一个持有锁的用户。所以（个人认为）<strong>如果仅仅从对资源的“排它”访问这一角度考虑，这里 token 的全局唯一确实等价于全局递增，但总的来说，全局递增更倾向于将交互权授予较新的锁持有者</strong>。</p>
<h3 id="Correctness-还是-Efficiency？"><a href="#Correctness-还是-Efficiency？" class="headerlink" title="Correctness 还是 Efficiency？"></a>Correctness 还是 Efficiency？</h3><p>看到这里，很容易产生一个误区，认为只要满足属性c，就能实现 Correctness。但事情没那么简单。假设我们的资源服务是某种 collection，对外提供添加元素与自增 size 的接口。我们希望使用分布式锁与 &quot;Fencing Token&quot; 机制来保证 collection 中元素数量与 size 属性的一致性。下面是一个可能的用户代码片段：<br><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="type">int</span> <span class="variable">token</span> <span class="operator">=</span> distributedLock.lock(collection, <span class="number">1000</span> * <span class="number">10</span>);  <span class="comment">// 锁的有效期为10秒</span></span><br><span class="line">collection.add(token, element);</span><br><span class="line">collection.incrementSize(token);</span><br><span class="line">distributedLock.unlock(token);</span><br></pre></td></tr></table></figure><br>该代码显然无法达成目的，考虑下面的执行序列：</p>
<ol>
<li>用户1获取一个 token 为 33 的锁，执行 <code>add</code> 添加元素，进程暂停，锁过期。</li>
<li>用户2获取一个 token 为 34 的锁，执行 <code>add</code> 与 <code>incrementSize</code> 成功。</li>
<li>用户1进程恢复，以为自己依然拥有锁，试图执行 <code>incrementSize</code>。</li>
<li>资源服务已经受理过 token=34 的请求，因此拒绝用户1请求，元素数量与 size 属性的一致性被破坏。</li>
</ol>
<p>通过分析这个例子，可以得出的结论是属性c依然无法满足 Correctness 需求，我们还需要：</p>
<blockquote>
<p>属性c&#39;：在任意时刻切换用户不会破坏资源服务的一致性。</p>
</blockquote>
<p>为了满足属性c&#39;，collection 服务至少有两种解决方案：</p>
<ul>
<li>方案一：像传统数据库对事务的处理那样，添加一个 commit, rollback 机制，如果在收到某个用户的 commit 请求之前发生了用户切换，rollback 该用户的所有操作。（这种方案需要某种机制来区分不同的用户，或者说不同的“事务”）</li>
<li>方案二：将 add 与 incrementSize 合并成一个原子操作。</li>
</ul>
<p>假设我们选择方案二，那么资源服务本身已经支持并发访问了，引入分布式锁服务只会降低整个系统的效率。<strong>假设我们选择方案一，那么分布式锁在这里的作用只是减少 rollback 的发生频率，即使不用 Fencing Token，最多也只是导致 rollback 发生得更频繁一点。更有甚者，即使不用锁，多个用户并发访问资源服务，导致资源服务频繁 rollback，没有任何一个用户能执行到 commit 逻辑，这也不会影响到资源服务数据的一致性。换句话说，这里使用分布式锁的目的已经不再是 Correctness，而是 Efficiency 了</strong>。</p>
<h3 id="结论"><a href="#结论" class="headerlink" title="结论"></a>结论</h3><p>尽管 HN 上大多数人都站在 Kleppmann 这边，认为 Redlock 只能用于 Efficiency 而不是 Correctness。但目前为止，我更偏向于支持 Sanfilippo，正如它文中提到的那样</p>
<blockquote>
<p>&quot;Most of the times when you need a distributed lock system that can guarantee mutual exclusivity, when this property is violated you already lost. Distributed locks are very useful exactly when we have no other control in the shared resource.&quot;</p>
</blockquote>
<p>更进一步，我认为<strong>所有带自动释放机制的分布式锁，本质上只是在为系统提供 Efficiency， 而 Correctness 需要资源服务自身通过某种机制（原子操作或事务日志等等）来保证</strong>。</p>
<h3 id="Reference"><a href="#Reference" class="headerlink" title="Reference"></a>Reference</h3><ul>
<li><a href="https://redis.io/topics/distlock">Redlock 介绍</a></li>
<li><a href="https://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html">How to do distributed locking</a></li>
<li><a href="http://antirez.com/news/101">Is Redlock safe?</a></li>
<li><a href="https://news.ycombinator.com/item?id=11059738">Hacker News 上对 Kleppmann 文章的讨论</a></li>
<li><a href="https://news.ycombinator.com/item?id=11065933">Hacker News 上对 Sanfilippo 文章的讨论</a></li>
</ul>
]]></content>
      
        <categories>
            
            <category> Backend </category>
            
        </categories>
        
        
        <tags>
            
            <tag> Distributed System </tag>
            
        </tags>
        
    </entry>
    
    <entry>
      <title><![CDATA[(译)Strong Consistency Models]]></title>
      <url>https://blog.staynoob.cn/post/2019/03/strong-consistency-model/</url>
      <content type="html"><![CDATA[<p><p style="text-align:center">（封面图片来自 <a href="https://jepsen.io/consistency">Consistency Models</a>）</p></p>
<blockquote>
<p>最近打算尝试一下翻译。由于我的英语基本停留在高中水平，所以不会严格按照原文来翻译，再加上我喜欢加入自己的理解（个人水平有限，所以我的理解应该也没啥参考价值）。所以有一定英语基础的同学还是建议自己阅读<a href="https://aphyr.com/posts/313-strong-consistency-models">原文：Strong Consistency Models</a>。</p>
</blockquote>
<h3 id="基础概念解释"><a href="#基础概念解释" class="headerlink" title="基础概念解释"></a>基础概念解释</h3><p>一些专业术语翻译成中文后往往更加难以理解，因此我不会翻译这些词，下面先简单解释一些本文中用得比较多的术语，其中的定义来自于 <a href="https://jepsen.io/consistency">Consistency Models</a> 这篇文章。这里只是做一个笼统的翻译。</p>
<ul>
<li>Systems<br>  分布式系统是一种并发 system，很多关于并发控制的研究可以直接应用到分布式 system 中。不过，大部分我们将要讨论的概念最开始是为单点并发系统设计的。它们之间在可用性和性能上还是有一些区别。<br>  System 的逻辑状态会随着时间改变。比如说单个整型变量就可以是一个简单的 system，它有类似于 0, 3, 42 这样的状态。一个互斥锁 system 有两种状态：locked 和 unlocked.</li>
<li>Operations<br>  一个 operation 是 system 从一种状态到另一种状态间的转移。比如说，一个单变量 system 可能有类似于读取和写入这样的 operation，它们分别用来获取和设置该变量的值。一个计数器可能有自增，自减，读取这样的 operation。</li>
<li>Histories<br>  一个 history 是一系列 operation 的集合，包括它们的并发结构。这里将其表述成一个包含 operation 的调用和完成的有序列表(an ordered list of invocation and completion operations)。</li>
<li>Consistency Models<br>  一个 consistency model 是一系列 history 的集合。我们用 consistency models 来定义哪些 histories 在 system 中是“好的”或者“合法的”。当我们说一个 history 违反了 serializability 或者不是 serializable 的时候，我们指的是这个 history 不在 serializable consistency model 允许的 history 集合。</li>
</ul>
<span id="more"></span>
<h3 id="正文"><a href="#正文" class="headerlink" title="正文"></a>正文</h3><p>由于网络分区总是会发生，交换机，网卡(Network Interface Controller, aka NIC)，主机硬件，操作系统，磁盘，抽象层，编程语言运行时，甚至程序语义自身，都在密谋着延迟，丢失，重复，或者乱排我们的消息，在一个不确定的世界，我们想要我们的软件维持直觉上的正确。但是什么是正确的事？我们怎样描述它？在这篇文章里，我们将会快速浏览一些“强”一致性模型，并且看看它们是如何在一起工作的。</p>
<h4 id="正确性-Correctness"><a href="#正确性-Correctness" class="headerlink" title="正确性(Correctness)"></a>正确性(Correctness)</h4><p>有很多种方式来表达一个算法的抽象行为，这里，我们说一个 system 是由 state 和一些改变状态的操作 operation 组成的。当 system 运行时，它经历一系列的操作(history of operations)来从某种状态，转移到另一种状态。<br><img src="/img/content/translation-strong-consistency-models/uniprocessor-history.jpg" alt="uniprocessor-history"><br>举例来说，我们的 state 可能是一个变量，operation 可能是写入或者读取该变量，在下面这个简单的 Ruby 程序中，我们多次写入和读取一个变量，并将其打印到控制台来演示读取操作。<br><figure class="highlight ruby"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">x = <span class="string">&quot;a&quot;</span>; puts x; puts x</span><br><span class="line">x = <span class="string">&quot;b&quot;</span>; puts x</span><br><span class="line">x = <span class="string">&quot;c&quot;</span></span><br><span class="line">x = <span class="string">&quot;d&quot;</span>; puts x</span><br></pre></td></tr></table></figure><br>对于程序的正确性，我们已经有一个直觉上的模型：它将打印 &quot;aabd&quot;. 为什么呢？因为这些语句按顺序执行。首先我们写入值 a，然后读取值 a，然后写入值 b，如此往复。<br>一旦我们将一个变量赋值为某个值，比如 a，读取该变量就应该返回 a，直到我们再次改变这个值。读取一个变量返回该变量最新写入的值。我们称这种 system （单个变量与单个值）为 register。<br>从我们第一天开始写程序起，这种模型就已经刻在我们脑海中了，就像自然本能一样。如果一个变量在读取的时候返回的是任意值: a, b, moon. 我们就会说这个 system 是不正确的，因为这些 history 跟我们设想的模型不一样。<br>这暗示着关于 system 正确的一个定义：给定一些关于 operations 和 state 的规则， system 中的任意 history of operations 应该一直遵守这些规则。我们称这些规则为一致性模型。<br>我们用文字来描述对 registers 的规则，但是这些规则也可能是任意复杂的数学结构。“一次读取返回该读取前倒数第二次写入的值，加上3，如果该值是4，这次读取就允许返回 cat 或者 dog”是一种一致性模型。“每次读取都返回0”也是一致性模型，甚至于“没有任何规则，所有的 history 都是允许的”也算，这算是最容易满足的一致性模型了，所有的 system 都遵守这个模型。<br>更严谨的来看，我们说一致性模型是所有允许的 histories 的集合。如果我们运行一个程序，我们会得到一个 history of operations，如果该 history 在允许的集合内，则这次运行是满足一致性要求的。如果一个程序时不时得到一个不在指定集合的 history，我们说这个 system 是不符合一致性要求的。如果所有可能得到的 history 最终都包含在该集合内，则这个 system 满足该模型。我们想要的是现实中的 system 满足“直觉上正确”的一致性模型，这样我们才能写出行为可预测的程序。</p>
<h4 id="Concurrent-histories"><a href="#Concurrent-histories" class="headerlink" title="Concurrent histories"></a>Concurrent histories</h4><blockquote>
<p>译注：注意下文的 process 跟我们平时理解的进程不一样，这里指的是一个逻辑上的单线程程序，一个 process 一次只做一件事。它的实现可能涉及到多线程，多进程，甚至多个物理节点，但只要这些组件最终能表现得像一个逻辑上的单线程程序就可以了。</p>
</blockquote>
<p>现在来设想一个并发程序，比如说一个 Node.js 或者 Erlang 程序。有多个逻辑线程，这里我们用术语 &quot;processes&quot; 来表述。如果我们执行一个有两个 processes 的并发程序，它们共同操作同一个 register。这可能会违反我们早先的不变性。<br><img src="/img/content/translation-strong-consistency-models/multiprocessor-history.jpg" alt="multiprocessor-history"><br>这里我们分别称两个工作的 processes 为 &quot;top&quot; 和 &quot;bottom&quot;。top process 尝试 write a，然后执行两次 read 操作。与此同时，bottom process 尝试 read，write b，再 read。因为程序是并发的，这两个 processes 的操作可能以多种方式排序。只要这些排序方式分别满足单个 process 指定的操作序列就可以了。<br>即使我们保证了对于单个 process 而言，operations 会按其指定的顺序执行，但我们先前对 register 的不变性定义依然被破坏了。top process 写入 a, 读到 a 值，接着又读到了 b ，这并不是它写入的值。因此，为了更好的定义并发，我们必须对我们直觉上的一致性模型做出修改。现在，一个 process 允许读取到任意 process 最近写入的值。该 register 变成了两个 process 协作的地方，它们共享 state。</p>
<h4 id="锥形光束-Light-cones"><a href="#锥形光束-Light-cones" class="headerlink" title="锥形光束(Light cones)"></a>锥形光束(Light cones)</h4><blockquote>
<p>译注：这里的光束(light)是一个假想模型，用来模拟消息随着时间推移从 process 抵达 register 并最终返回所经过的路径。</p>
</blockquote>
<p>现实还不止于此，在几乎所有真实的 system 中，processes 之间是存在距离的。比如说，内存中一个未缓存的值，在 DIMM(Dual Inline Memory Module) 中可能距离 CPU 30厘米远。我们的 light 需要花费整整一纳秒的时间来走过这段距离（真实的内存访问会更加慢）。一个值可能存在于几千米外的某个数据中心的某台机器上，一条消息在“路上”可能需要花费几百毫秒的时间，目前来说，我们从物理上改变不了这个事实。<br><img src="/img/content/translation-strong-consistency-models/lightcone-history.jpg" alt="lightcone-history"><br>这意味着我们的 operation 不再是立即发生的，有些时候它们快到我们可以忽略不计，但是总的来说，operations 需要在路上花费一些时间。大致的情形是：我们执行一个写指令，该指令到达内存，或另一台计算机，或者月球；内存修改状态；发回一条确认消息，告诉我们这个 operation 的确执行成功了。<br>消息从一个地方到另一个地方之间的延误，预示着同样的 history of operations 可能会导致不同的执行结果。消息到达的快慢可能导致它们实际的执行顺序跟我们期望的不一样。这里，bottom process 在值是 a 的时候调用读操作，当该消息还未到达目的地的时候，top process 调用写入 b，恰巧这个写入操作比读取操作先到达。最后 bottom process 在值为 a 的时候读取，最终却读到了 b。<br><img src="/img/content/translation-strong-consistency-models/concurrent-read.jpg" alt="concurrent-read"><br>这个 history 再次违背了我们现有的 concurrent register consistency model。bottom process 在调用读操作的时候并没有读取到当前值。也许你会说我们可以用操作的完成时间(completion time)，而不是调用时间(invocation time)来作为 operation 的真实时间。同样的道理，该结果依然不正确，因为如果读操作比写操作先到达，bottom process 会收到 a 值，但实际的值确是 b。<br>在分布式系统中（一个 operation 需要时间来到达生效点），我们需要再次放宽我们的一致性模型，来允许这些带有歧义的顺序发生。</p>
<h4 id="线性一致性-Linearizability，又称-atomic-consistency"><a href="#线性一致性-Linearizability，又称-atomic-consistency" class="headerlink" title="线性一致性(Linearizability，又称 atomic consistency)"></a>线性一致性(Linearizability，又称 atomic consistency)</h4><p>仔细思考一下，上文的事件发生顺序还是有界的。一条消息到达数据源的时间不可能早于消息的发出时间，所以一个 operation 不可能在其调用之前就生效。同理，收到执行成功的确认消息的时间也必定晚于 operation 生效的时间，也就是说 operation 不可能在完成时间之后才生效。<br><img src="/img/content/translation-strong-consistency-models/finite-concurrency-bounds.jpg" alt="finite-concurrency-bounds"><br>如果我们假设所有的 processes 都与一个全局的状态对话，并且所有 operation 对该状态中心的影响是原子级的，对彼此之间没有依赖。在这些规则下我们可以排除掉很多可能的 histories。我们知道每个 operation 都会在它的调用时间与返回时间之间的某个时间点生效。<br>我们称这种一致性模型为线性一致性，因为尽管 operations 是并发的，而且需要时间来完成，但是它们总是有可能能保持一个合理的线性顺序的。<br>线性一致性是一种非常强壮的模型，一旦一个 operation 完成，所有人都能看到它的结果（或者一些更新的结果），因为所有的 operation 都会在它完成前生效，并且后续调用的 operation 肯定会在调用的时间点后生效。这就意味着一旦我们成功的写入了 b，所有后续的读取操作都会看到 b（或者如果发生了其它的写入，我们会看到比 b 更新的值）。<br><img src="/img/content/translation-strong-consistency-models/linearizability-complete-visibility.jpg" alt="linearizability-complete-visibility"><br>我们可以利用线性一致性的原子级别约束来安全的修改(mutate) state。我们可以定义一些类似于 compare-and-set 的 operation，即当且仅当当前 register 处于某个值的时候，将其设置为新的某个值。我们可以用 compare-and-set 作为实现 mutexes, semaphores, channels, counters, lists, sets maps, trees 的基础，所有的共享数据结构都变得可行了，线性一致性能够保证我们安全的更新这些数据。<br>不止于此，线性一致性保证了在一个 operation 完成后，其修改结果对其它的参与者可见。因此它可以防止 stale reads，即保证每个 operation 都能读取到处于调用时间与完成时间之间的某个值，而不是调用前的值。同时它也可以防止 non-monotonic reads，即保证后续的读取操作只可能读取到更新的值。<br>因为这些约束，满足线性一致性的 systems 非常容易理解以及预测，这也是为什么它被选择作为很多并发编程结构的基础。在 Javascript 中所有的变量都是满足线性一致性的，同样还有 Java 中的 volatile 变量，Clojure 中的 atoms，Erlang 中的独立 processes。大多数语言都有 mutexes 和 semophores 实现，这些也是 linearizable 的。<br>但是如果我们不能满足这些约束会发生什么呢？</p>
<h4 id="顺序一致性-Sequential-consistency"><a href="#顺序一致性-Sequential-consistency" class="headerlink" title="顺序一致性(Sequential consistency)"></a>顺序一致性(Sequential consistency)</h4><p>如果我们允许 processes 倾斜时间，比如说允许 operations 在调用前就生效，或者在完成后才生效，但是对任意 process，来自该 process 的 operations 必须按其指定的顺序生效。这样我们就得到了一个相比之下稍弱的一致性模型，即 sequential consistency.<br><img src="/img/content/translation-strong-consistency-models/sequential-history.jpg" alt="sequential-history"><br>Sequential consistency 相比线性一致性允许更多的 histories。但是它仍然是一个有用的模型：我们每天都在使用它。比如说当一个用户上传一部视频到 youtube 的时候，youtube 将该视频存入一个队列，等待后续的处理，然后直接返回一个网页。我们在当时无法直接看到该视频，视频上传行为需要等视频处理完后才算生效。这里的队列移除了同步行为，但保留了正确的顺序。<br>很多缓存实现也像 sequentially consistent systems。如果我在 Twitter 上发一条 tweet。它需要花一些时间来渗透各种缓存层。不同的用户看到这条消息的时间可能不一样，但是每个用户始终会按照顺序看到我们的 operations。一旦看到了，它不会再消失，如果我发了多条评论，它们也会按顺序变得可见。</p>
<blockquote>
<p>译注: 注意上文中的“倾斜时间”，“调用前生效”，“完成后生效”可能从字面上很难理解，这里补充一个例子。比如在 Java 程序中一个可以被多个线程访问的变量（注意没有 <code>volatile</code> 关键字，也没有加锁），其中有一条语句是修改这个变量，这条语句执行完意味着已经过了 completion time，但是实际上有可能该 operation 并没有真的“生效”。因为其它线程可能并不能看到该修改结果。</p>
</blockquote>
<h4 id="因果一致性-Causal-consistency"><a href="#因果一致性-Causal-consistency" class="headerlink" title="因果一致性(Causal consistency)"></a>因果一致性(Causal consistency)</h4><p>我们有可能并不需要保持一个 process 所有的 operation 都有序，有时只要因果相关的 operation 有序就行了。如果我们对每个 operation 都存入一条类似于“我依赖于 operation X”这样的信息，数据源就可以延迟 operation ，直到它依赖的所有 operation 都生效为止。<br>这种模型比 Sequential consistency 要弱，在这种模型中，即便对于同一个 process 来说，不相关的 operation 也可能以任意顺序执行。但由于互相依赖的 operation 依然有序执行，因此依然阻止了很多违反直觉的行为。</p>
<h4 id="Serializable-consistency"><a href="#Serializable-consistency" class="headerlink" title="Serializable consistency"></a>Serializable consistency</h4><p>如果我们说任意 history of operations 一定等价于某种原子排序(If we say that the history of operations is equivalent to one that took place in some single atomic order)，但对 operation 的调用与完成时间不作出任何承诺。我们就获得了一种被称为 serializability 的一致性模型。这种模型跟你的预期相比，既强壮得多，也弱小得多。<br><img src="/img/content/translation-strong-consistency-models/serializable-history.jpg" alt="serializable-history"><br>Serializability 可以非常的弱，因为它对时间与排序都不设限制，在上面的图中，它表现得就像一个消息可以传回任意远的过去或未来，因果关系也允许被破坏。在一个 serializable 的数据库中，一个类似于 read x 这样的事务允许在 time 为 0 的时刻执行，也可能那时 x 还没有被初始化，或者它也可能被延迟到无穷远的未来！事务 write 2 to x 可能现在就执行，也可能被延迟到永远也不会执行。<br>比如说在 serializable system 中，下面这个程序<br><figure class="highlight ruby"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">x = <span class="number">1</span></span><br><span class="line">x = x + <span class="number">1</span></span><br><span class="line">puts x</span><br></pre></td></tr></table></figure><br>打印 nil, 1, 或者 2 都是允许的，因为 operations 可能以任意排序生效，这是一种非常弱的约束，我们假设每行代码代表一个 operation ，并且所有 operation 都成功执行了。<br>另一方面，serializability 模型又非常强。下面的程序<br><figure class="highlight ruby"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">print x <span class="keyword">if</span> x = <span class="number">3</span></span><br><span class="line">x = <span class="number">1</span> <span class="keyword">if</span> x = <span class="literal">nil</span></span><br><span class="line">x = <span class="number">2</span> <span class="keyword">if</span> x = <span class="number">1</span></span><br><span class="line">x = <span class="number">3</span> <span class="keyword">if</span> x = <span class="number">2</span></span><br></pre></td></tr></table></figure><br>只有一种排序方式，虽然它不会按我们写的顺序执行，但是它能够可靠的将 x 按顺序修改成 nil, 1, 2, 3。并且最终打印3。<br>因为 serializability 允许将 operations 任意排序，因此在真实的应用中这种模型并没有什么用，很多数据库声称提供 serializability 模型，但实际提供的是 strong serializability 模型。这种模型有着跟 linearizability 一样的时间界限。更糟糕的是，很多 SQL 数据库中所谓的 SERIALIZABLE consistency level 实际上用的是某种更弱的模型，比如说 repeatable read, cursor stablity, 或者 snapshot isolation。</p>
<h4 id="一致性的代价"><a href="#一致性的代价" class="headerlink" title="一致性的代价"></a>一致性的代价</h4><p>我们说过“弱”的一致性模型相比“强”一致性模型允许更多的 histories。比方说 Linearizability 模型，向我们保证了 operations 会在调用时间与完成时间之间的某个时间点生效。但是维护这样的秩序是需要协作的。大致来说，我们排除的 histories 越多，系统的参与者所需要的沟通就越多，并且需要更加谨慎。<br>你也许已经听过了 CAP 理论，它说的是任意 system 最多只能保证实现 consistency, availability 和 partition tolerance 的其中两个。更精确的定义是：</p>
<ol>
<li>Consistency 意味着 Linearizability。</li>
<li>Availability 意味着对任意一个在线节点的请求必须在有限时间内成功的完成。因为我们允许网络分区持续无限久，这意味这我们不能简单的将回复推迟到网络分区结束。</li>
<li>Partition tolerance 意味着允许网络分区发生。当网络可靠时，保持 consistency 和 availability 是很容易的。当网络不可靠时，同时保证这两者已经被证明了是不可能的。如果你的网络不是完美可靠的（它本来就不可能是），你就不能选择 CA，这意味着所有实践中的分布式系统最多只能保证 AP 或者 CP。</li>
</ol>
<p>“等等！”，也许你会说。“Linearizability 并不是唯一的一致性模型，为了绕过 CAP 定理，我还可以提供 sequential，或者 serailizability，或者 snapshot isolation 一致性！”<br>没错，CAP 定理只说了我们无法构建百分百可用的 linearizable systems。问题在于还有其它的证明告诉了我们你也无法构建百分百可用的 sequential, serializable, repeatable read, snapshot isolation 或者 cursor stability 或者任意比这些模型 &quot;strong&quot; 的一致性模型。在下图中（该图等价于封面图，所展示的理论来自于<a href="http://www.vldb.org/pvldb/vol7/p181-bailis.pdf">Highly Available Transactions paper</a>），红色的模型都是无法完全可用的。<br><img src="/img/content/translation-strong-consistency-models/family-tree.jpg" alt="family-tree"><br>如果我们稍微放松一些对 availability 的要求，比如说客户端永远只会与相同的服务端节点会话，这样的话其中一些模型是可以实现的，比如说 causal consistency, PRAM(Pipelined RAM, also FIFO consistency), 和 read-your-writes 一致性。<br>如果我们要求百分百可用，我们可能实现的是 monotonic reads, monotonic writes, read committed, monotonic atomic view 等等。一些分布式数据库提供了这些模型，比如说 Riak 和 Cassandra，或者被设置成低隔离级别的 ANSI SQL 数据库。这些模型不像我们之前在图中画的那样线性有序，它们只提供部分有序 (partial oerder)。</p>
<h4 id="一种混合方式"><a href="#一种混合方式" class="headerlink" title="一种混合方式"></a>一种混合方式</h4><p><img src="/img/content/translation-strong-consistency-models/weak-not-unsafe.jpg" alt="weak-not-unsafe"><br>一些算法需要 linearizability 来保证其正确性。比如说，如果我们想构建一个分布式锁服务，没有时间界限，我们可能会持有一把来自过去或者未来的锁。另一方面，也有很多算法不需要 linearizability。比如说，一些可以被表示成 CRDTs(Convergent and Commutative<br>Replicated Data Types) 的最终一致性集合只需要&quot;弱&quot;的一致性模型就可以实现。<br>更强壮的一致性模型往往倾向于需要更多的协作，即跟多的消息交换，来保证它们的 operations 出现在正确的顺序。它们不仅可用性更低，而且可能导致更高的延迟，这也是为什么默认情况下，现代的 CPU 内存模型不是 linearizable 的。除非你显式的指明，不然的话现代的 CPU 可能会打乱相对其它核心的内存 operations 的顺序，或者更糟。这么做对整体性能的提升是现象级的。一些在地理上不同节点相距甚远的分布式系统，相互间的沟通可能会有上百毫秒的延迟，这样的系统也会经常做类似的妥协。<br>因此，在实际场景中，我们经常使用混合多种一致性模型的数据存储方案来完成我们对 redundancy, availability, performance, 和 safety objectives 的需求。为了可用性和性能考虑，只要条件允许，我们优先选择“弱”一致性模型。在必要时才选择“强”一致性模型。比如说你可以将大容量的数据写入 S3, Riak 或者 Cassandra，然后将它们的引用线性化的写入 Postgres, Zookeeper 或者 Etcd。一些数据库实现了多种一致性模型，比如关系型数据库中的隔离级别是可调的，或者 Cassandra 和 Riak 提供 linearizable 事务。最后：任何声称他们的一致性模型是唯一正确选择的人，很有可能只是在做产品推销。强一致性，可用性和性能之间，始终存在一种“鱼和熊掌不可兼得”的关系。</p>
<h3 id="补充说明"><a href="#补充说明" class="headerlink" title="补充说明"></a>补充说明</h3><p>下面的链接是一些翻译过程中阅读过的文章或资料：</p>
<ul>
<li><a href="https://jepsen.io/consistency">Consistency Models</a></li>
<li><a href="https://en.wikipedia.org/wiki/Consistency_model">Consistency model(wikipedia)</a></li>
<li><a href="http://www.bailis.org/blog/linearizability-versus-serializability/">Linearizability vs Serializability</a></li>
<li><a href="https://youtu.be/hUd_9FENShA?si=7EzSgeUB-_HfO-zQ">You don’t need CP, you don’t want AP, and you can’t have CA</a></li>
</ul>
]]></content>
      
        <categories>
            
            <category> Backend </category>
            
        </categories>
        
        
        <tags>
            
            <tag> Distributed System </tag>
            
        </tags>
        
    </entry>
    
    <entry>
      <title><![CDATA[Common Pitfalls in JPA(Hibernate)]]></title>
      <url>https://blog.staynoob.cn/post/2019/02/common-pitfalls-in-jpa-hibernate/</url>
      <content type="html"><![CDATA[<blockquote>
<p>Nowadays, ORM technique has been playing an important role in object-oriented programming, and JPA is now considered the standard industry approach for ORM in the Java industry. In this post, I summarized several phenomena which violate my intuition and prone to error.</p>
</blockquote>
<p>As JPA itself is just a specification, there are various underlying implementation. In this post, we are only focusing on Hibernate implementation. In fact, I&#39;ve never used or tested any other implementation so far, which means there&#39;s a chance that a problem cannot be reproduced in other JPA implementation.</p>
<h3 id="Prerequisites"><a href="#Prerequisites" class="headerlink" title="Prerequisites"></a>Prerequisites</h3><p>As in post <a href="/post/2019/02/common-pitfalls-of-declarative-transaction-management-in-spring/">Common Pitfalls of Declarative Transaction Management in Spring</a>, all the samples are written in Kotlin language. And Spring Data JPA framework is used for the sake of convenience. Full source code can be found at <a href="https://github.com/noob9527/post-code/tree/master/2-common-pitfalls-in-jpa-hibernate">common-pitfalls-in-jpa-hibernate</a>.</p>
<h3 id="Pitfall-1-Don-39-t-be-fooled-by-equals-and-hashcode-methods"><a href="#Pitfall-1-Don-39-t-be-fooled-by-equals-and-hashcode-methods" class="headerlink" title="Pitfall 1: Don&#39;t be fooled by equals and hashcode methods"></a>Pitfall 1: Don&#39;t be fooled by <code>equals</code> and <code>hashcode</code> methods</h3><p>You may already know that there are several contracts we have to obey when implementing <code>equals</code> and <code>hashcode</code> method. Namely Reflexivity, Symmetry, Transitivity, Consistency and &quot;Non-nullity&quot;. When it comes to a JPA entity, things become even more difficult since entity state transitions must be taken into account. In other words, <code>equals</code> and <code>hashcode</code> methods must behave consistently across all entity state transitions. Thus, we can immediately conclude that <strong> logical key(usually auto generate after the first time being persisted) should not be taken into consideration. </strong> <code>AbstractPersistable</code> from spring data JPA library is a perfect counterexample which implements <code>equals</code> and <code>hashcode</code> based on auto-generation id. The following code demonstrates its flaw:<br><figure class="highlight kotlin"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">@Entity</span></span><br><span class="line"><span class="keyword">class</span> <span class="title class_">Demo1</span> : <span class="type">AbstractPersistable</span>&lt;<span class="type">Int</span>&gt;()</span><br></pre></td></tr></table></figure><br><figure class="highlight kotlin"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">@Test</span></span><br><span class="line"><span class="function"><span class="keyword">fun</span> <span class="title">test</span><span class="params">()</span></span> &#123;</span><br><span class="line">    <span class="keyword">val</span> demo = Demo1()</span><br><span class="line">    <span class="keyword">val</span> <span class="keyword">set</span> = hashSetOf(demo)</span><br><span class="line"></span><br><span class="line">    <span class="keyword">set</span>.contains(demo) <span class="comment">// true</span></span><br><span class="line"></span><br><span class="line">    entityManager.persist(demo)</span><br><span class="line">    entityManager.flush()</span><br><span class="line"></span><br><span class="line">    <span class="keyword">set</span>.contains(demo) <span class="comment">// false</span></span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br>The <code>HashSet</code> failed to recognize the same entity since its hashcode changed after being persisted. Certainly, this is error-prone. For similar reason, <strong> default <code>equals</code> and <code>hashcode</code> inherited from <code>java.lang.Object</code> is not suitable for JPA entity either. </strong> Code below shows that a merged entity isn&#39;t equal to itself because <code>entityManager.merge</code> may return a different object reference.<br><figure class="highlight kotlin"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">@Entity</span></span><br><span class="line"><span class="keyword">class</span> <span class="title class_">Demo2</span> : <span class="type">Persistable</span>&lt;<span class="type">Int</span>&gt; &#123;</span><br><span class="line">    <span class="meta">@Id</span></span><br><span class="line">    <span class="meta">@GeneratedValue</span></span><br><span class="line">    <span class="keyword">private</span> <span class="keyword">var</span> id: <span class="built_in">Int</span>? = <span class="literal">null</span></span><br><span class="line"></span><br><span class="line">    <span class="keyword">override</span> <span class="function"><span class="keyword">fun</span> <span class="title">getId</span><span class="params">()</span></span>: <span class="built_in">Int</span>? &#123;</span><br><span class="line">        <span class="keyword">return</span> id</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    <span class="keyword">override</span> <span class="function"><span class="keyword">fun</span> <span class="title">isNew</span><span class="params">()</span></span>: <span class="built_in">Boolean</span> &#123;</span><br><span class="line">        <span class="keyword">return</span> id == <span class="literal">null</span></span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    <span class="comment">// inherit equals and hashcode from Object</span></span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br><figure class="highlight kotlin"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">@Test</span></span><br><span class="line"><span class="function"><span class="keyword">fun</span> <span class="title">test</span><span class="params">()</span></span> &#123;</span><br><span class="line">    <span class="keyword">val</span> demo = Demo2()</span><br><span class="line">    <span class="keyword">val</span> <span class="keyword">set</span> = hashSetOf(demo)</span><br><span class="line"></span><br><span class="line">    <span class="keyword">set</span>.contains(demo) <span class="comment">// true</span></span><br><span class="line"></span><br><span class="line">    entityManager.persist(demo)</span><br><span class="line">    entityManager.flush()</span><br><span class="line">    entityManager.detach(demo)</span><br><span class="line"></span><br><span class="line">    <span class="keyword">val</span> managed = entityManager.merge(demo)</span><br><span class="line"></span><br><span class="line">    <span class="keyword">set</span>.contains(managed) <span class="comment">// false</span></span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br>Now, the only option left to us is implementing <code>equals</code> and <code>hashcode</code> methods based on some business key, and never change the key after the entity is created. However, you can not always find such keys in practical. In such cases, the best we can do is no matter which way we choose to implement the methods, be aware of its shortcomings and document them clearly.</p>
<p>Reference:</p>
<ul>
<li><a href="https://vladmihalcea.com/a-beginners-guide-to-jpa-hibernate-entity-state-transitions/">A beginner’s guide to entity state transitions with JPA and Hibernate</a></li>
<li><a href="https://vladmihalcea.com/hibernate-facts-equals-and-hashcode/">How to implement Equals and HashCode for JPA entities</a></li>
<li><a href="https://vladmihalcea.com/how-to-implement-equals-and-hashcode-using-the-jpa-entity-identifier/">How to implement equals and hashCode using the JPA entity identifier (Primary Key)</a></li>
</ul>
<span id="more"></span>
<h3 id="Pitfall-2-LazyCollectionOption-EXTRA-can-make-your-collection-behave-unpredictably"><a href="#Pitfall-2-LazyCollectionOption-EXTRA-can-make-your-collection-behave-unpredictably" class="headerlink" title="Pitfall 2: LazyCollectionOption.EXTRA can make your collection behave unpredictably"></a>Pitfall 2: <code>LazyCollectionOption.EXTRA</code> can make your collection behave unpredictably</h3><p>A collection can be lazy fetch or eager fetch in JPA. Eager fetching is generally considered as an <a href="https://vladmihalcea.com/eager-fetching-is-a-code-smell/">anti-pattern</a>, so we are not going to talk about it. Lazy fetch is preferred since it improves performance by reducing unnecessary queries. In hibernate, we can make a lazy collection even more &quot;lazy&quot; by putting a <code>@LazyCollection(LazyCollectionOption.EXTRA)</code> annotation on the collection. After doing so, when <code>collection.size</code> is called, hibernate will fire a select count query instead of loading the whole collection. This sounds very practical let alone we can implement it by just adding one line of code. The following code shows how an &quot;extra lazy&quot; collection looks like:<br><figure class="highlight kotlin"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">@Entity</span></span><br><span class="line"><span class="keyword">class</span> <span class="title class_">Parent</span> : <span class="type">AbstractPersistable</span>&lt;<span class="type">Int</span>&gt;() &#123;</span><br><span class="line"></span><br><span class="line">    <span class="meta">@OneToMany(mappedBy = <span class="string">&quot;parent&quot;</span>, cascade = [CascadeType.ALL], orphanRemoval = true)</span></span><br><span class="line">    <span class="meta">@LazyCollection(LazyCollectionOption.EXTRA)</span></span><br><span class="line">    <span class="keyword">val</span> children: MutableSet&lt;Child&gt; = mutableSetOf()</span><br><span class="line"></span><br><span class="line">    <span class="comment">// ignore other methods...</span></span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br>The problem is that not only it optimizes the <code>collection.size</code> call, it also affects the behavior of <code>collection.contains</code>. If a collection is fully loaded, the <code>contains</code> method behave like a usual collection, which means the result depends on <code>equals</code> or <code>hashcode</code> method of its elements. Otherwise, it will send a SQL query based on the primary key of the given element instead of loading the whole collection. That&#39;s why I said the collection could behave unpredictably. To make this concrete, let&#39;s say our child entity has a unique business key &quot;name&quot;, and we implement the <code>equals</code> and <code>hashcode</code> base on that key as we recommended.<br><figure class="highlight kotlin"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">@Entity</span></span><br><span class="line"><span class="meta">@Table(uniqueConstraints = [UniqueConstraint(columnNames = [<span class="string">&quot;name&quot;</span>])])</span></span><br><span class="line"><span class="keyword">class</span> <span class="title class_">Child</span>(</span><br><span class="line">        <span class="meta">@Column(unique = true)</span></span><br><span class="line">        <span class="keyword">val</span> name: String,</span><br><span class="line"></span><br><span class="line">        <span class="meta">@ManyToOne</span></span><br><span class="line">        <span class="meta">@JoinColumn</span></span><br><span class="line">        <span class="keyword">val</span> parent: Parent</span><br><span class="line">) : AbstractPersistable&lt;<span class="built_in">Int</span>&gt;() &#123;</span><br><span class="line">    <span class="keyword">override</span> <span class="function"><span class="keyword">fun</span> <span class="title">equals</span><span class="params">(other: <span class="type">Any</span>?)</span></span>: <span class="built_in">Boolean</span> &#123;</span><br><span class="line">        <span class="keyword">if</span> (<span class="keyword">this</span> === other) <span class="keyword">return</span> <span class="literal">true</span></span><br><span class="line">        <span class="keyword">if</span> (javaClass != other?.javaClass) <span class="keyword">return</span> <span class="literal">false</span></span><br><span class="line"></span><br><span class="line">        other <span class="keyword">as</span> Child</span><br><span class="line"></span><br><span class="line">        <span class="keyword">if</span> (name != other.name) <span class="keyword">return</span> <span class="literal">false</span></span><br><span class="line"></span><br><span class="line">        <span class="keyword">return</span> <span class="literal">true</span></span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    <span class="keyword">override</span> <span class="function"><span class="keyword">fun</span> <span class="title">hashCode</span><span class="params">()</span></span>: <span class="built_in">Int</span> &#123;</span><br><span class="line">        <span class="keyword">var</span> result = <span class="number">17</span></span><br><span class="line">        result = <span class="number">31</span> * result + name.hashCode()</span><br><span class="line">        <span class="keyword">return</span> result</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br>Then we may encounter the awkward situation below, <code>children.contains(child)</code> return inconsistent result depends on if the collection is loaded.<br><figure class="highlight kotlin"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">val</span> tmp = Parent()</span><br><span class="line">        .apply &#123;</span><br><span class="line">            addChild(Child(<span class="string">&quot;test&quot;</span>, <span class="keyword">this</span>))</span><br><span class="line">        &#125;</span><br><span class="line">entityManager.persist(tmp)</span><br><span class="line">entityManager.flush()</span><br><span class="line">entityManager.clear()</span><br><span class="line"></span><br><span class="line"><span class="keyword">val</span> parent = entityManager.find(Parent::<span class="keyword">class</span>.java, tmp.id)</span><br><span class="line"><span class="keyword">val</span> child = Child(<span class="string">&quot;test&quot;</span>, tmp)</span><br><span class="line"></span><br><span class="line">println(parent.children.contains(child))    <span class="comment">// false</span></span><br><span class="line">Hibernate.initialize(parent.children)</span><br><span class="line">println(parent.children.contains(child))    <span class="comment">// true</span></span><br></pre></td></tr></table></figure><br>If <code>collection.contains()</code> method behaves unpredictable, the uniqueness contract of a set element will also be broken. You should be totally aware of these consequence before you decide to make a collection &quot;extra&quot; lazy.</p>
<h3 id="Pitfall-3-Statement-may-not-be-executed-in-the-order-in-which-it-is-written-during-flushing"><a href="#Pitfall-3-Statement-may-not-be-executed-in-the-order-in-which-it-is-written-during-flushing" class="headerlink" title="Pitfall 3: Statement may not be executed in the order in which it is written(during flushing)"></a>Pitfall 3: Statement may not be executed in the order in which it is written(during flushing)</h3><p>Let&#39;s say we have the same model from the preceding example. To focus on this problem, we remove the <code>@LazyCollection</code> annotation from the children collection and change the child entity to have a composite unique key so that the child name uniqueness is limited in certain parent range.<br><figure class="highlight kotlin"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">@Entity</span></span><br><span class="line"><span class="keyword">class</span> <span class="title class_">Parent</span> : <span class="type">AbstractPersistable</span>&lt;<span class="type">Int</span>&gt;() &#123;</span><br><span class="line"></span><br><span class="line">    <span class="meta">@OneToMany(mappedBy = <span class="string">&quot;parent&quot;</span>, cascade = [CascadeType.ALL], orphanRemoval = true)</span></span><br><span class="line">    <span class="comment">// @LazyCollection(LazyCollectionOption.EXTRA)</span></span><br><span class="line">    <span class="keyword">val</span> children: MutableSet&lt;Child&gt; = mutableSetOf()</span><br><span class="line"></span><br><span class="line">    <span class="comment">// ignore other methods...</span></span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="meta">@Entity</span></span><br><span class="line"><span class="meta">@Table(uniqueConstraints = [UniqueConstraint(columnNames = [<span class="string">&quot;name&quot;</span>, <span class="string">&quot;parent_id&quot;</span>])])</span> <span class="comment">// note the &quot;parent_id&quot; column</span></span><br><span class="line"><span class="keyword">class</span> <span class="title class_">Child</span>(</span><br><span class="line">        <span class="keyword">val</span> name: String,</span><br><span class="line"></span><br><span class="line">        <span class="meta">@ManyToOne</span></span><br><span class="line">        <span class="meta">@JoinColumn</span></span><br><span class="line">        <span class="keyword">val</span> parent: Parent</span><br><span class="line">) : AbstractPersistable&lt;<span class="built_in">Int</span>&gt;()</span><br></pre></td></tr></table></figure><br>Now, suppose we need to update the children of a persisted parent, we may end up with writing code like this<br><figure class="highlight kotlin"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">val</span> child = Child(<span class="string">&quot;foo&quot;</span>, parent);</span><br><span class="line">parent.children.clear()</span><br><span class="line">parent.addChild(child)</span><br><span class="line">entityManager.persist(parent)</span><br><span class="line">entityManager.flush()</span><br></pre></td></tr></table></figure><br>This code works fine in most case. However, if the parent already has a child called &quot;foo&quot;, we&#39;ll encounter an <code>ConstraintViolationException</code>. Even though we&#39;ve already removed all the existed children before adding. The reason is hibernate doesn&#39;t execute statements in the order in which the code is written during flushing. Instead, it has its own <a href="http://docs.jboss.org/hibernate/orm/4.2/javadocs/org/hibernate/event/internal/AbstractFlushingEventListener.html#performExecutions%28org.hibernate.event.spi.EventSource%29">defined order</a>, which says inserts always happen before deletes. So the same error can also be reproduced in another way:<br><figure class="highlight kotlin"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">val</span> parent1 = Parent(<span class="string">&quot;foo&quot;</span>)</span><br><span class="line"><span class="keyword">val</span> parent2 = Parent(<span class="string">&quot;foo&quot;</span>)</span><br><span class="line"></span><br><span class="line">entityManager.persist(parent1)</span><br><span class="line">entityManager.flush()</span><br><span class="line"></span><br><span class="line">entityManager.remove(parent1)</span><br><span class="line"><span class="comment">// to avoid the issue, we have to call flush here.</span></span><br><span class="line"><span class="comment">// otherwise the ConstraintViolationException will be thrown</span></span><br><span class="line"><span class="comment">// since hibernate will try to insert before deleting.</span></span><br><span class="line"><span class="comment">// entityManager.flush()</span></span><br><span class="line">entityManager.persist(parent2)</span><br><span class="line"></span><br><span class="line">entityManager.flush()  <span class="comment">// throw ConstraintViolationException</span></span><br></pre></td></tr></table></figure><br>As the code comments, we have to call addition <code>entity.flush()</code> after performing the delete action. Certainly it violates our intuition.<br>Reference:</p>
<ul>
<li><a href="https://stackoverflow.com/questions/17410868/hibernate-jpa-onetomany-delete-old-insert-new-without-flush">Hibernate JPA: @OneToMany delete old, insert new without flush</a></li>
</ul>
<h3 id="Pitfall-4-PreUpdate-hook-may-not-be-invoked-as-you-think"><a href="#Pitfall-4-PreUpdate-hook-may-not-be-invoked-as-you-think" class="headerlink" title="Pitfall 4: PreUpdate hook may not be invoked as you think"></a>Pitfall 4: <code>PreUpdate</code> hook may not be invoked as you think</h3><p>This time, We set up a <code>preUpdate</code> hook function to automatically record how many times an instance has been changed since it was persisted, and we also change the <code>children</code> property to be mutable for demonstrating sake.<br><figure class="highlight kotlin"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">@Entity</span></span><br><span class="line"><span class="keyword">class</span> <span class="title class_">Parent</span> : <span class="type">AbstractPersistable</span>&lt;<span class="type">Int</span>&gt;() &#123;</span><br><span class="line"></span><br><span class="line">    <span class="meta">@OneToMany(mappedBy = <span class="string">&quot;parent&quot;</span>, cascade = [CascadeType.ALL])</span></span><br><span class="line">    <span class="keyword">var</span> children: MutableSet&lt;Child&gt; = mutableSetOf()</span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">fun</span> <span class="title">addChild</span><span class="params">(child: <span class="type">Child</span>)</span></span> &#123;</span><br><span class="line">        children.add(child)</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">fun</span> <span class="title">removeChild</span><span class="params">(child: <span class="type">Child</span>)</span></span> &#123;</span><br><span class="line">        children.remove(child)</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    <span class="keyword">var</span> updateTimes: <span class="built_in">Int</span> = <span class="number">0</span></span><br><span class="line">        <span class="keyword">private</span> <span class="keyword">set</span></span><br><span class="line"></span><br><span class="line">    <span class="meta">@PreUpdate</span></span><br><span class="line">    <span class="function"><span class="keyword">fun</span> <span class="title">preUpdate</span><span class="params">()</span></span> &#123;</span><br><span class="line">        updateTimes++</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br>Then you&#39;ll see that mutate children <strong>collection</strong> won&#39;t trigger the hook, whereas mutating children <strong>property</strong> will. Because the dirty check mechanism won&#39;t detect the collection mutation.<br><figure class="highlight kotlin"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">val</span> parent = Parent()</span><br><span class="line"></span><br><span class="line">entityManager.persist(parent)</span><br><span class="line">entityManager.flush()</span><br><span class="line"></span><br><span class="line">println(parent.updateTimes) <span class="comment">// 0</span></span><br><span class="line"></span><br><span class="line">parent.children = mutableSetOf(Child(<span class="string">&quot;test1&quot;</span>, parent))</span><br><span class="line">entityManager.merge(parent)</span><br><span class="line">entityManager.flush()</span><br><span class="line"></span><br><span class="line">println(parent.updateTimes) <span class="comment">// 1</span></span><br><span class="line"></span><br><span class="line">parent.children.add(Child(<span class="string">&quot;test2&quot;</span>, parent))</span><br><span class="line">entityManager.merge(parent)</span><br><span class="line">entityManager.flush()</span><br><span class="line"></span><br><span class="line">println(parent.updateTimes) <span class="comment">// 1</span></span><br></pre></td></tr></table></figure><br>From the framework&#39;s perspective, this makes sense in a way, because detecting elements modification of a collection might be too expensive in such a case. But from a user&#39;s perspective, you may want to increase the <code>updateTimes</code> value whenever a model is updated.</p>
<h3 id="Pitfall-5-Delete-action-may-be-discarded-silently"><a href="#Pitfall-5-Delete-action-may-be-discarded-silently" class="headerlink" title="Pitfall 5: Delete action may be discarded silently"></a>Pitfall 5: Delete action may be discarded silently</h3><p>Again, with the same parent, child model. If you want to remove a child via <code>entityManager.remove</code> method, it simply doesn&#39;t work. The following code exhibits this phenomenon.<br><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line">val parent = Parent()</span><br><span class="line">val child = Child(&quot;test&quot;, parent)</span><br><span class="line">parent.addChild(child)</span><br><span class="line"></span><br><span class="line">entityManager.persist(parent)</span><br><span class="line">entityManager.flush()</span><br><span class="line"></span><br><span class="line">entityManager.remove(child) // will be discarded silently</span><br><span class="line">entityManager.flush()</span><br><span class="line"></span><br><span class="line">val res = entityManager.find(Child::class.java, child.id)</span><br><span class="line"></span><br><span class="line">println(res == null)    // false</span><br></pre></td></tr></table></figure><br>The reason is we have a cascade relation between parent and child entity. Either we remove the cascade property from <code>@OneToMany</code> annotation, or we have to ensure that the same entity is already removed from the <code>parent.children</code> collection. Otherwise, the delete action will be silently discarded as if the <code>entityManager.remove(child)</code> has never been called.</p>
<h3 id="Conclusion"><a href="#Conclusion" class="headerlink" title="Conclusion"></a>Conclusion</h3><p>JPA does a great job of saving programmers from writing SQL by hand, but write correct code with high performance in JPA may not as easy as you thought. Hopefully, this post can save you from making the same mistakes as I made.</p>
]]></content>
      
        <categories>
            
            <category> Backend </category>
            
        </categories>
        
        
        <tags>
            
            <tag> JPA </tag>
            
            <tag> Spring </tag>
            
        </tags>
        
    </entry>
    
    <entry>
      <title><![CDATA[Common Pitfalls of Declarative Transaction Management in Spring]]></title>
      <url>https://blog.staynoob.cn/post/2019/02/common-pitfalls-of-declarative-transaction-management-in-spring/</url>
      <content type="html"><![CDATA[<blockquote>
<p>Spring supports two types of transaction management, namely, programmatic and declarative transaction management. Despite the fact that programmatic management is more flexible, declarative management is still preferred since it is less invasive to application code. In this post, I&#39;m going to summarize several pitfalls you may encounter while using declarative transaction management. Certainly, if you read the official document thoroughly, you should know how to avoid them on your own, but if you think of it is all about annotating your method with the @Transactional annotation as I did, you may never figure them out until the day your customer reports his balance is incorrect.</p>
</blockquote>
<h3 id="Prerequisites"><a href="#Prerequisites" class="headerlink" title="Prerequisites"></a>Prerequisites</h3><p>Our samples are written in Kotlin language. In addition, I assume that you are already familiar with the following frameworks.</p>
<ul>
<li>Spring</li>
<li>JPA(Hibernate implementation)</li>
<li>Spring Data JPA</li>
</ul>
<p>Examples in this post are based on the following class:<br><figure class="highlight kotlin"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// entity</span></span><br><span class="line"><span class="meta">@Entity</span></span><br><span class="line"><span class="keyword">class</span> <span class="title class_">DemoEntity</span>(</span><br><span class="line">        <span class="keyword">val</span> name: String</span><br><span class="line">) : AbstractPersistable&lt;<span class="built_in">Int</span>&gt;()</span><br><span class="line"></span><br><span class="line"><span class="comment">// repository</span></span><br><span class="line"><span class="keyword">interface</span> <span class="title class_">DemoRepo</span> : <span class="type">CrudRepository</span>&lt;<span class="type">DemoEntity, Int</span>&gt;</span><br></pre></td></tr></table></figure><br>Read the manual of Spring Data JPA If you are not able to understand the above code. Finally, The related test code will be available at <a href="https://github.com/noob9527/post-code/tree/master/1-common-pitfalls-of-declarative-transaction-management-in-spring">common-pitfalls-of-declarative-transaction-management-in-spring</a>.</p>
<h3 id="Pitfall-1-Transactional-annotation-may-have-no-effect-at-all"><a href="#Pitfall-1-Transactional-annotation-may-have-no-effect-at-all" class="headerlink" title="Pitfall 1: @Transactional annotation may have no effect at all"></a>Pitfall 1: @Transactional annotation may have no effect at all</h3><p>It&#39;s a common circumstance that we put some code in a private method so it can be reused. If the code involves a transaction, we may end up with writing code like this.<br><figure class="highlight kotlin"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">@Service</span></span><br><span class="line"><span class="keyword">class</span> <span class="title class_">DemoService</span>(</span><br><span class="line">        <span class="keyword">private</span> <span class="keyword">val</span> demoRepo: DemoRepo</span><br><span class="line">) &#123;</span><br><span class="line">    <span class="function"><span class="keyword">fun</span> <span class="title">persistAndDoSomething</span><span class="params">(demo: <span class="type">DemoEntity</span>)</span></span> &#123;</span><br><span class="line">        persist(demo)</span><br><span class="line">        <span class="comment">// do something</span></span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">fun</span> <span class="title">persistAndDoOtherthings</span><span class="params">(demo: <span class="type">DemoEntity</span>)</span></span> &#123;</span><br><span class="line">        persist(demo)</span><br><span class="line">        <span class="comment">// do other things</span></span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    <span class="meta">@Transactional</span></span><br><span class="line">    <span class="keyword">private</span> <span class="function"><span class="keyword">fun</span> <span class="title">persist</span><span class="params">(demo: <span class="type">DemoEntity</span>)</span></span> &#123;</span><br><span class="line">        <span class="comment">// you may think this action will be rolled back if exception occurs</span></span><br><span class="line">        demoRepo.save(demo)</span><br><span class="line">        unpredictableMethod()</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    <span class="comment">// simulate a method which may or may not throw an exception</span></span><br><span class="line">    <span class="function"><span class="keyword">fun</span> <span class="title">unpredictableMethod</span><span class="params">()</span></span> &#123;</span><br><span class="line">        <span class="keyword">if</span> (ThreadLocalRandom.current().nextBoolean())</span><br><span class="line">            <span class="keyword">throw</span> Exception(<span class="string">&quot;Oops!&quot;</span>)</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br>In this case, the <code>persist</code> method will be invoked as if no @Transactional annotation is present. To understand why, we need to know that the declarative transaction is implemented on top of AOP(Aspect Oriented Programming) proxies. A proxied method invocation procedure looks like this:<br><img src="/img/content/common-pitfalls-of-declarative-transaction-management-in-spring/transactional-proxy.png" alt="transactional-proxy"><br>From the picture, it is not hard to imagine that the <code>beginTransaction</code>, <code>commit</code> and <code>rollback</code> logic is implemented in a so called &quot;advice&quot; component. &quot;Advice&quot; here refers to a core concept of AOP(read the documentation of <a href="https://docs.spring.io/spring/docs/4.2.x/spring-framework-reference/html/aop.html">Spring AOP</a> to get more information), and there is two different advice mode supported by Spring transaction management, which called &quot;PROXY&quot; and &quot;ASPECTJ&quot;. As the document says:</p>
<blockquote>
<p>When using proxies, you should apply the @Transactional annotation only to methods with public visibility. If you do annotate protected, private or package-visible methods with the @Transactional annotation, no error is raised, but the annotated method does not exhibit the configured transactional settings. Consider the use of AspectJ if you need to annotate non-public methods.</p>
</blockquote>
<p>Now the reason is pretty clear, to fix the problem, we can either switch the advice mode from &quot;PROXY&quot;(default option) to &quot;ASPECTJ&quot;, or remove the private modifier from the <code>persist</code> method. Let&#39;s say we choose to remove the modifier, you can find that the <code>persist</code> is still invoked without any transaction, because we just fall into the next pitfall.</p>
<span id="more"></span>
<h3 id="Pitfall-2-Transactional-annotation-may-not-affect-a-method-invocation"><a href="#Pitfall-2-Transactional-annotation-may-not-affect-a-method-invocation" class="headerlink" title="Pitfall 2: @Transactional annotation may not affect a method invocation"></a>Pitfall 2: @Transactional annotation may not affect a method invocation</h3><p>Following the previous section, after applying the second solution from the previous section, our <code>DemoService</code> now looks like:<br><figure class="highlight kotlin"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">@Service</span></span><br><span class="line"><span class="keyword">class</span> <span class="title class_">DemoService</span>(</span><br><span class="line">        <span class="keyword">private</span> <span class="keyword">val</span> demoRepo: DemoRepo</span><br><span class="line">) &#123;</span><br><span class="line">    <span class="function"><span class="keyword">fun</span> <span class="title">persistAndDoSomething</span><span class="params">(demo: <span class="type">DemoEntity</span>)</span></span> &#123;</span><br><span class="line">        persist(demo)</span><br><span class="line">        <span class="comment">// do something</span></span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">fun</span> <span class="title">persistAndDoOtherthings</span><span class="params">(demo: <span class="type">DemoEntity</span>)</span></span> &#123;</span><br><span class="line">        persist(demo)</span><br><span class="line">        <span class="comment">// do other things</span></span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    <span class="meta">@Transactional</span></span><br><span class="line">    <span class="function"><span class="keyword">fun</span> <span class="title">persist</span><span class="params">(demo: <span class="type">DemoEntity</span>)</span></span> &#123;</span><br><span class="line">        <span class="comment">// you may think this action will be rolled back if exception occurs</span></span><br><span class="line">        demoRepo.save(demo)</span><br><span class="line">        unpredictableMethod()</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    <span class="comment">// simulate a method which may or may not throw an exception</span></span><br><span class="line">    <span class="function"><span class="keyword">fun</span> <span class="title">unpredictableMethod</span><span class="params">()</span></span> &#123;</span><br><span class="line">        <span class="keyword">if</span> (ThreadLocalRandom.current().nextBoolean())</span><br><span class="line">            <span class="keyword">throw</span> Exception(<span class="string">&quot;Oops!&quot;</span>)</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br>This time, if you inject the instance of <code>DemoService</code> to another class, then directly call <code>demoService.persist(demo)</code> from that class, everything works. It can mislead you to think of the <code>persist</code> method is already being proxied whereas it is not. Again, the official document has its own explanation for this problem.</p>
<blockquote>
<p>In proxy mode, only external method calls coming in through the proxy are intercepted. This means that self-invocation, in effect, a method within the target object calling another method of the target object, will not lead to an actual transaction at runtime even if the invoked method is marked with <code>@Transactional</code>. Also, the proxy must be fully initialized to provide the expected behavior so you should not rely on this feature in your initialization code, i.e. <code>@PostConstruct</code>.</p>
</blockquote>
<p>There&#39;s a tricky solution against this problem, inject itself to an instance field, then invoke methods from the instance. The code looks like<br><figure class="highlight kotlin"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">@Service</span></span><br><span class="line"><span class="keyword">class</span> <span class="title class_">DemoService</span>(</span><br><span class="line">        <span class="keyword">private</span> <span class="keyword">val</span> demoRepo: DemoRepo</span><br><span class="line">) &#123;</span><br><span class="line"></span><br><span class="line">    <span class="comment">// Unlike demoRepo, it cannot be a constructor parameter</span></span><br><span class="line">    <span class="comment">// otherwise Spring won&#x27;t be able to initialize the proxy due to a circular dependency.</span></span><br><span class="line">    <span class="meta">@Autowired</span></span><br><span class="line">    <span class="keyword">private</span> lazyinit <span class="keyword">var</span> self: DemoService</span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">fun</span> <span class="title">persistAndDoSomething</span><span class="params">(demo: <span class="type">DemoEntity</span>)</span></span> &#123;</span><br><span class="line">        self.persist(demo)</span><br><span class="line">        <span class="comment">// do something</span></span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">fun</span> <span class="title">persistAndDoOtherthings</span><span class="params">(demo: <span class="type">DemoEntity</span>)</span></span> &#123;</span><br><span class="line">        self.persist(demo)</span><br><span class="line">        <span class="comment">// do other things</span></span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    <span class="meta">@Transactional</span></span><br><span class="line">    <span class="function"><span class="keyword">fun</span> <span class="title">persist</span><span class="params">(demo: <span class="type">DemoEntity</span>)</span></span> &#123;</span><br><span class="line">        <span class="comment">// you may think this action will be rolled back if exception occurs</span></span><br><span class="line">        demoRepo.save(demo)</span><br><span class="line">        unpredictableMethod()</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    <span class="comment">// simulate a method which may or may not throw an exception</span></span><br><span class="line">    <span class="function"><span class="keyword">fun</span> <span class="title">unpredictableMethod</span><span class="params">()</span></span> &#123;</span><br><span class="line">        <span class="keyword">if</span> (ThreadLocalRandom.current().nextBoolean())</span><br><span class="line">            <span class="keyword">throw</span> Exception(<span class="string">&quot;Oops!&quot;</span>)</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br>Finally, the <code>persist</code> method is now invoked with a transaction, but somehow, the transaction still does not rollback no matter whether the exception is thrown. Guess what, we again fall into the next pitfall.</p>
<h3 id="Pitfall-3-Transaction-may-not-rollback-when-an-exception-occurs"><a href="#Pitfall-3-Transaction-may-not-rollback-when-an-exception-occurs" class="headerlink" title="Pitfall 3: Transaction may not rollback when an exception occurs"></a>Pitfall 3: Transaction may not rollback when an exception occurs</h3><p>Before further explaining, let&#39;s focus on the buggy code segment.<br><figure class="highlight kotlin"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">@Transactional</span></span><br><span class="line"><span class="function"><span class="keyword">fun</span> <span class="title">persist</span><span class="params">(demo: <span class="type">DemoEntity</span>)</span></span> &#123;</span><br><span class="line">    <span class="comment">// you may think this action will be rolled back if exception occurs</span></span><br><span class="line">    demoRepo.save(demo)</span><br><span class="line">    unpredictableMethod()</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="comment">// simulate a method which may or may not throw an exception</span></span><br><span class="line"><span class="function"><span class="keyword">fun</span> <span class="title">unpredictableMethod</span><span class="params">()</span></span> &#123;</span><br><span class="line">    <span class="keyword">if</span> (ThreadLocalRandom.current().nextBoolean())</span><br><span class="line">        <span class="keyword">throw</span> Exception(<span class="string">&quot;Oops!&quot;</span>)</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br>Providing only ten lines of code, an experienced Spring developer may realize, &quot;The exception here, is it a runtime exception?&quot;. Yes, that&#39;s the point. By default, Spring won&#39;t mark a transaction for rollback after any exception is thrown. As the document says:</p>
<blockquote>
<p>In its default configuration, the Spring Framework’s transaction infrastructure code only marks a transaction for rollback in the case of runtime, unchecked exceptions; that is, when the thrown exception is an instance or subclass of RuntimeException. ( Errors will also - by default - result in a rollback). Checked exceptions that are thrown from a transactional method do not result in rollback in the default configuration.</p>
</blockquote>
<p>Admittedly, the hidden bug would be much more obvious if I wrote the code segment in Java language, because there isn&#39;t any difference between checked exception and runtime exception in Kotlin. In Java language, on the contrary, the exception must be handled or throw to the method caller explicitly, that could make this problem easier to identify. Nevertheless, I&#39;ve made this kind of mistake while I was using Java, That&#39;s why I think it&#39;s worth to mention.<br>Now let&#39;s turn to the solution. According to the official document, we can just configure the <code>@Transactional</code> annotation to rollback for any exception. Or, we can catch the exception at the first place.<br><figure class="highlight kotlin"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">@Transactional</span></span><br><span class="line"><span class="function"><span class="keyword">fun</span> <span class="title">persist</span><span class="params">(demo: <span class="type">DemoEntity</span>)</span></span> &#123;</span><br><span class="line">    demoRepo.save(demo)</span><br><span class="line"></span><br><span class="line">    <span class="keyword">try</span> &#123;</span><br><span class="line">        unpredictableMethod()</span><br><span class="line">    &#125; <span class="keyword">catch</span> (e: Exception) &#123;</span><br><span class="line">        <span class="comment">// handle the exception</span></span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br>This solution seems a little tedious to an experienced programmer, But by doing so, we get an opportunity to run into next pitfall.</p>
<h3 id="Pitfall-4-Transaction-may-not-be-able-to-commit-after-an-exception-is-being-caught"><a href="#Pitfall-4-Transaction-may-not-be-able-to-commit-after-an-exception-is-being-caught" class="headerlink" title="Pitfall 4: Transaction may not be able to commit after an exception is being caught"></a>Pitfall 4: Transaction may not be able to commit after an exception is being caught</h3><p>After hours of tinkering, we finally get the piece of code work, But requirements are changing, let&#39;s say somehow we need to ensure the <code>unpredictableMethod</code> run with a transaction. After going through previous lessons, this time we want to do it extremely carefully.<br>Firstly, we put a <code>@Transactional</code> on the method and configure it to rollback for all exceptions, so that we won&#39;t fall into pitfall 3.<br><figure class="highlight kotlin"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">@Transactional(rollbackFor=[Exception::class])</span></span><br><span class="line"><span class="function"><span class="keyword">fun</span> <span class="title">unpredictableMethod</span><span class="params">()</span></span> &#123;</span><br><span class="line">    <span class="comment">/*</span></span><br><span class="line"><span class="comment">        some business code must be executed with a transaction</span></span><br><span class="line"><span class="comment">        ...</span></span><br><span class="line"><span class="comment">    */</span></span><br><span class="line">    <span class="keyword">if</span> (ThreadLocalRandom.current().nextBoolean())</span><br><span class="line">        <span class="keyword">throw</span> Exception(<span class="string">&quot;Oops!&quot;</span>)</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br>Secondly, Owing to pitfall 2, we update the self-invocation in <code>persist</code> method to the proxied method call.<br><figure class="highlight kotlin"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">@Transactional</span></span><br><span class="line"><span class="function"><span class="keyword">fun</span> <span class="title">persist</span><span class="params">(demo: <span class="type">DemoEntity</span>)</span></span> &#123;</span><br><span class="line">    demoRepo.save(demo)</span><br><span class="line"></span><br><span class="line">    <span class="keyword">try</span> &#123;</span><br><span class="line">        self.unpredictableMethod()</span><br><span class="line">    &#125; <span class="keyword">catch</span> (e: Exception) &#123;</span><br><span class="line">        <span class="comment">// handle the exception</span></span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br>After everything is checked, we expect that if <code>unpredictableMethod</code> is called directly and an exception is thrown, the transaction will be rolled back. Conversely, if <code>persist</code> is called, the transaction will be committed since exceptions should be handled by the <code>try-catch</code> block. Unfortunately, it is not happening, we just ran into pitfall 4. An <code>UnexpectedRollbackException</code> will be thrown.<br>When a &quot;transactional&quot; method invokes another &quot;transactional&quot; method, the actual behavior is determined by <code>propagation</code> property of the second <code>@Transaction</code> annotation. If the property value is <code>REQUIRED</code>, which is the default value, the second method will be executed in an existing transaction, and once the rollback condition in the second method is reached, it marks the transaction as rollback-only. That is, no matter if the exception is handled in the first method, the transaction won&#39;t be able to commit. If you like, there&#39;s an official explanation:</p>
<blockquote>
<p>When the propagation setting is PROPAGATION_REQUIRED, a logical transaction scope is created for each method upon which the setting is applied. Each such logical transaction scope can determine rollback-only status individually, with an outer transaction scope being logically independent from the inner transaction scope. Of course, in case of standard PROPAGATION_REQUIRED behavior, all these scopes will be mapped to the same physical transaction. So a rollback-only marker set in the inner transaction scope does affect the outer transaction’s chance to actually commit (as you would expect it to).<br>However, in the case where an inner transaction scope sets the rollback-only marker, the outer transaction has not decided on the rollback itself, and so the rollback (silently triggered by the inner transaction scope) is unexpected. A corresponding UnexpectedRollbackException is thrown at that point. This is expected behavior so that the caller of a transaction can never be misled to assume that a commit was performed when it really was not. So if an inner transaction (of which the outer caller is not aware) silently marks a transaction as rollback-only, the outer caller still calls commit. The outer caller needs to receive an UnexpectedRollbackException to indicate clearly that a rollback was performed instead.</p>
</blockquote>
<p>The solution depends on what you want, you can change the <code>propagation</code> property of the second <code>@Transactional</code>, or change the <code>rollbackFor</code> and <code>noRollbackFor</code> property of the first <code>@Transactional</code> to filter a specific exception class. In this case, you can even fix the problem by falling into the pitfall 2 deliberately, although it is not recommended because your colleague may not be able to understand it.</p>
<h3 id="Conclusion"><a href="#Conclusion" class="headerlink" title="Conclusion"></a>Conclusion</h3><p>Spring declarative transaction management does hide the complexity of writing transaction handling code, but handle transaction correctly is far more complex than just put an <code>@Transactional</code> annotation on your method. Considering transaction is usually the most critical part in an application, you&#39;d better test every branch of your code.</p>
]]></content>
      
        <categories>
            
            <category> Backend </category>
            
        </categories>
        
        
        <tags>
            
            <tag> JPA </tag>
            
            <tag> Spring </tag>
            
        </tags>
        
    </entry>
    
    <entry>
      <title><![CDATA[Understanding Zombie Process]]></title>
      <url>https://blog.staynoob.cn/post/2019/01/understanding-zombie-process/</url>
      <content type="html"><![CDATA[<blockquote>
<p>As a programmer, I usually feel uncomfortable when <code>top</code> command reports there&#39;re zombie processes running on my computer. After some study, I found that the zombie process is not as scary as my thought. This article briefly introduces the zombie process in UNIX-like systems.</p>
</blockquote>
<h3 id="What-is-a-quot-zombie-process-quot"><a href="#What-is-a-quot-zombie-process-quot" class="headerlink" title="What is a &quot;zombie process&quot;?"></a>What is a &quot;zombie process&quot;?</h3><blockquote>
<p>&quot;In UNIX System terminology, a process that has terminated, but whose parent has not yet waited for it, is called a zombie.&quot;</p>
</blockquote>
<p>After we create a process via <code>fork</code> function, we get a parent process and a child process. The parent process sometimes needs to know how the child is terminated. In normal cases, we call <code>wait</code> or <code>waitpid</code> to fetch the termination status. However, a child process could terminate before its parent waits for it. In such a case, If the system cleared the child&#39;s information completely, its parent wouldn&#39;t be able to know its status. As a result, the kernel has to keep a small amount of information after a process terminates. A process like this that has been terminated, but not completely disappear, is called a zombie process.</p>
<blockquote>
<p>Note that zombie processes should not be confused with orphan processes: an orphan process is a process that is still executing, but whose parent has died. These do not remain as zombie processes; instead, (like all orphaned processes) they are adopted by <code>init</code>, which waits on its children. The result is that a process that is both a zombie and an orphan will be reaped automatically.</p>
</blockquote>
<span id="more"></span>
<h3 id="How-to-produce-a-quot-zombie-process-quot"><a href="#How-to-produce-a-quot-zombie-process-quot" class="headerlink" title="How to produce a &quot;zombie process&quot;?"></a>How to produce a &quot;zombie process&quot;?</h3><p>Before we create a zombie process, we need to know how to identify a zombie process. The <code>ps</code> command prints the state of a zombie process as Z. We can also fetch the state of a process with the command below<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="built_in">cat</span> /proc/&#123;pid&#125;/status | grep state</span><br></pre></td></tr></table></figure><br>In our example, we run above command programmatically to print the state of a process, the function will look like:<br><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="type">int</span> <span class="title function_">print_process_state</span><span class="params">(<span class="type">pid_t</span> pid)</span> &#123;</span><br><span class="line">    <span class="type">char</span> cmdstring[<span class="number">80</span>];</span><br><span class="line">    <span class="type">char</span> *end = cmdstring;</span><br><span class="line">    end += <span class="built_in">sprintf</span>(end, <span class="string">&quot;%s&quot;</span>, <span class="string">&quot;cat /proc/&quot;</span>);</span><br><span class="line">    end += <span class="built_in">sprintf</span>(end, <span class="string">&quot;%ld&quot;</span>, (<span class="type">long</span>) pid);</span><br><span class="line">    <span class="built_in">sprintf</span>(end, <span class="string">&quot;%s&quot;</span>, <span class="string">&quot;/status | grep State&quot;</span>);</span><br><span class="line"></span><br><span class="line">    <span class="built_in">puts</span>(cmdstring);</span><br><span class="line">    system(cmdstring);</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br>The example is pretty straightforward, we <code>fork</code> a process, then let parent process sleep one second so that the child process will terminate first.<br><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"><span class="type">int</span> <span class="title function_">main</span><span class="params">(<span class="type">void</span>)</span> &#123;</span><br><span class="line">    <span class="type">pid_t</span> pid;</span><br><span class="line"></span><br><span class="line">    <span class="keyword">if</span> ((pid = fork()) &gt; <span class="number">0</span>) &#123;</span><br><span class="line">        sleep(<span class="number">1</span>); <span class="comment">// let child terminates first</span></span><br><span class="line">        <span class="comment">// print the state of child process</span></span><br><span class="line">        print_process_state(pid); <span class="comment">// =&gt; Zombie</span></span><br><span class="line">    &#125; <span class="keyword">else</span> <span class="keyword">if</span> (pid == <span class="number">0</span>) &#123;</span><br><span class="line">        <span class="comment">// this process will become a zombie process after return statement</span></span><br><span class="line">        <span class="built_in">printf</span>(<span class="string">&quot;child proceess pid = %ld\n&quot;</span>, (<span class="type">long</span>) getpid());</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    <span class="keyword">return</span> <span class="number">0</span>;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure></p>
<h3 id="How-to-avoid-producing-quot-zombie-process-quot"><a href="#How-to-avoid-producing-quot-zombie-process-quot" class="headerlink" title="How to avoid producing &quot;zombie process&quot;?"></a>How to avoid producing &quot;zombie process&quot;?</h3><p>Although the kernel will only keep a small amount of information for a zombie process, We should still avoid producing them, there&#39;re several ways to accomplish this.</p>
<h4 id="wait-and-waitpid-functions"><a href="#wait-and-waitpid-functions" class="headerlink" title="wait and waitpid functions"></a><code>wait</code> and <code>waitpid</code> functions</h4><p>The most trivial way to avoid zombie process is to call wait function after a child process terminates. However, the wait function will block the caller until a child process terminates. If you don&#39;t want your program to be blocked, you can call wait function in a <code>SIGCHLD</code> signal handler. The wait function will return immediately if there&#39;s a child process is waiting for its status to be fetched.<br><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br></pre></td><td class="code"><pre><span class="line"><span class="type">static</span> <span class="type">void</span> <span class="title function_">sig_chld_handler</span><span class="params">(<span class="type">int</span> signo)</span> &#123;</span><br><span class="line">    <span class="keyword">if</span> (signo == SIGCHLD) &#123;</span><br><span class="line">        <span class="type">pid_t</span> pid = wait(<span class="literal">NULL</span>);</span><br><span class="line">        <span class="keyword">if</span> (pid &gt; <span class="number">0</span>) <span class="built_in">printf</span>(<span class="string">&quot;received SIGCHLD from child process pid = %d\n&quot;</span>, pid);</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="comment">/**</span></span><br><span class="line"><span class="comment"> * avoid producing zombie processing by calling one of the wait functions</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"><span class="type">int</span> <span class="title function_">main</span><span class="params">(<span class="type">void</span>)</span> &#123;</span><br><span class="line">    <span class="type">pid_t</span> pid;</span><br><span class="line"></span><br><span class="line">    signal(SIGCHLD, sig_chld_handler);</span><br><span class="line"></span><br><span class="line">    <span class="keyword">if</span> ((pid = fork()) &gt; <span class="number">0</span>) &#123;</span><br><span class="line">        sleep(<span class="number">2</span>); <span class="comment">// let child terminates first</span></span><br><span class="line">        print_process_state(pid);   <span class="comment">// prints error since the child process is completely cleared</span></span><br><span class="line">    &#125; <span class="keyword">else</span> <span class="keyword">if</span> (pid == <span class="number">0</span>) &#123;</span><br><span class="line">        <span class="comment">// this process won&#x27;t become a zombie process</span></span><br><span class="line">        <span class="built_in">printf</span>(<span class="string">&quot;child process pid = %ld\n&quot;</span>, (<span class="type">long</span>) getpid());</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    <span class="keyword">return</span> <span class="number">0</span>;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure></p>
<h4 id="Explicitly-ignore-SIGCHLD-signal"><a href="#Explicitly-ignore-SIGCHLD-signal" class="headerlink" title="Explicitly ignore SIGCHLD signal"></a>Explicitly ignore <code>SIGCHLD</code> signal</h4><p>Although the <code>SIGCHLD</code> signal is ignored by default, you have to ignore it explicitly to tell the kernel you really don&#39;t care how your children processes are terminated, so there&#39;s no need to keep any information of a terminated child process, which means there&#39;s no need to keep a zombie process.<br><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="type">int</span> <span class="title function_">main</span><span class="params">(<span class="type">void</span>)</span> &#123;</span><br><span class="line">    signal(SIGCHLD, SIG_IGN);</span><br><span class="line">    <span class="comment">// fork code</span></span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure></p>
<h4 id="Call-fork-twice"><a href="#Call-fork-twice" class="headerlink" title="Call fork twice"></a>Call <code>fork</code> twice</h4><p>Knowing that a child process will be inherited by <code>init</code> process if its parent is terminated, and the <code>init</code> process is written so that whenever its children terminate, it calls one of the wait function to fetch the termination status. We can avoid zombie process by only creating &quot;orphan process&quot;, the trick is call fork twice and have the first child terminated directly.<br><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">/**</span></span><br><span class="line"><span class="comment"> * avoid producing zombie processing by calling fork twice</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"><span class="type">int</span> <span class="title function_">main</span><span class="params">(<span class="type">void</span>)</span> &#123;</span><br><span class="line">    <span class="type">pid_t</span> pid;</span><br><span class="line"></span><br><span class="line">    <span class="keyword">if</span> ((pid = fork()) == <span class="number">0</span>) &#123;</span><br><span class="line">        <span class="comment">/* first child */</span></span><br><span class="line">        <span class="keyword">if</span> (fork() &gt; <span class="number">0</span>) &#123;</span><br><span class="line">            <span class="comment">/* parent from second fork == first child */</span></span><br><span class="line">            <span class="built_in">exit</span>(<span class="number">0</span>);</span><br><span class="line">        &#125;</span><br><span class="line"></span><br><span class="line">        <span class="comment">/*</span></span><br><span class="line"><span class="comment">         * We&#x27;re the second child; our parent becomes init as soon</span></span><br><span class="line"><span class="comment">         * as our real parent calls exit() in the statement above.</span></span><br><span class="line"><span class="comment">         * Here&#x27;s where we&#x27;d continue executing, knowing that when</span></span><br><span class="line"><span class="comment">         * we&#x27;re done, init will reap our status.</span></span><br><span class="line"><span class="comment">         */</span></span><br><span class="line">        sleep(<span class="number">2</span>);</span><br><span class="line">        <span class="built_in">printf</span>(<span class="string">&quot;second child, parent pid = %ld\n&quot;</span>, (<span class="type">long</span>) getppid());</span><br><span class="line">        <span class="built_in">exit</span>(<span class="number">0</span>);</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    waitpid(pid, <span class="literal">NULL</span>, <span class="number">0</span>);    <span class="comment">/* wait for first child */</span></span><br><span class="line"></span><br><span class="line">    <span class="comment">/*</span></span><br><span class="line"><span class="comment">     * We&#x27;re the parent (the original process); we continue executing,</span></span><br><span class="line"><span class="comment">     * knowing that we&#x27;re not the parent of the second child.</span></span><br><span class="line"><span class="comment">     */</span></span><br><span class="line">    <span class="keyword">return</span> <span class="number">0</span>;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure></p>
<h3 id="Reference"><a href="#Reference" class="headerlink" title="Reference"></a>Reference</h3><ul>
<li>Advanced Programming in the UNIX Environment(AKA APUE)</li>
<li><a href="https://en.wikipedia.org/wiki/Zombie\_process">Zombie process</a></li>
</ul>
]]></content>
      
        <categories>
            
            <category> System </category>
            
        </categories>
        
        
        <tags>
            
            <tag> UNIX </tag>
            
        </tags>
        
    </entry>
    
    <entry>
      <title><![CDATA[再谈js闭包]]></title>
      <url>https://blog.staynoob.cn/post/2017/07/%25E5%2586%258D%25E8%25B0%2588js%25E9%2597%25AD%25E5%258C%2585/</url>
      <content type="html"><![CDATA[<blockquote>
<p>网上关于 js 闭包的文章多如牛毛，这里之所以再写一篇，主要是因为网上的那些文章要么对初学者不够友好，要么根本就没有谈到重点。在读过那些文章后的很长一段时间里，我对闭包都是似懂非懂。直到在学 react 的过程中逐渐接触函数式编程，才开始真正理解闭包。</p>
</blockquote>
<h3 id="Trick"><a href="#Trick" class="headerlink" title="Trick"></a>Trick</h3><p>请先思考一下下面两段代码：<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">function</span> <span class="title function_">createFunction</span>(<span class="params"></span>) &#123;</span><br><span class="line">    <span class="keyword">let</span> arr = [];</span><br><span class="line"></span><br><span class="line">    <span class="keyword">for</span> (<span class="keyword">var</span> i = <span class="number">0</span>; i &lt; <span class="number">5</span>; i++) &#123;</span><br><span class="line">        arr[i] = <span class="keyword">function</span> (<span class="params"></span>) &#123;</span><br><span class="line">            <span class="keyword">return</span> i;</span><br><span class="line">        &#125;;</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    <span class="keyword">return</span> arr;</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="keyword">const</span> result = <span class="title function_">createFunction</span>().<span class="title function_">map</span>(<span class="function"><span class="params">e</span> =&gt;</span> <span class="title function_">e</span>());</span><br><span class="line"></span><br><span class="line"><span class="variable language_">console</span>.<span class="title function_">log</span>(result);    <span class="comment">// [5, 5, 5, 5, 5]</span></span><br></pre></td></tr></table></figure><br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">function</span> <span class="title function_">counter</span>(<span class="params"></span>) &#123;</span><br><span class="line">    <span class="keyword">let</span> n = <span class="number">0</span>;</span><br><span class="line">    <span class="keyword">return</span> &#123;</span><br><span class="line">        <span class="title function_">increase</span>(<span class="params"></span>) &#123; ++n; &#125;,</span><br><span class="line">        <span class="title function_">get</span>(<span class="params"></span>) &#123; <span class="keyword">return</span> n; &#125;,</span><br><span class="line">    &#125;;</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="keyword">const</span> cnt1 = <span class="title function_">counter</span>();</span><br><span class="line"><span class="keyword">const</span> cnt2 = <span class="title function_">counter</span>();</span><br><span class="line"></span><br><span class="line">cnt1.<span class="title function_">increase</span>();</span><br><span class="line"></span><br><span class="line"><span class="variable language_">console</span>.<span class="title function_">log</span>(cnt1.<span class="title function_">get</span>());    <span class="comment">// 1</span></span><br><span class="line"><span class="variable language_">console</span>.<span class="title function_">log</span>(cnt2.<span class="title function_">get</span>());    <span class="comment">// 0</span></span><br></pre></td></tr></table></figure></p>
<p>在往下阅读之前，请再次确保你有花时间理解上面两段代码。虽然它们跟下文内容并没有一毛钱关系，不过反正没事烧烧脑也没什么坏处。。。<br>这里我想表达的只是，网上大量关于闭包的文章大抵都遵循这个模式，先制造一堆跟上面例子类似的函数，之后让读者尝试给出运行结果，最后在配合上自己的一顿讲解，仿佛能理解这些代码就是懂了闭包。而事实却是，能看懂这些代码并不代表你就理解了闭包，理解闭包之后再看这些代码也不一定就都能立刻指出运行结果。</p>
<span id="more"></span>
<h3 id="Definition"><a href="#Definition" class="headerlink" title="Definition"></a>Definition</h3><blockquote>
<p>函数对象可以通过作用域链相互关联起来，函数体内部的变量都可以保存在函数作用域内，这种特性在计算机科学文献中称为“闭包”。从技术的角度讲，所有的javascript函数都是闭包</p>
</blockquote>
<!-- -->
<blockquote>
<p>闭包是指有权访问另一个函数作用域中的变量的函数</p>
</blockquote>
<p>上面两段话分别摘自中译本的《javascript权威指南》和《javascript高级程序设计》，不知道你看懂了没，反正我是看不懂。（不过依然强烈推荐 js 初学者选择这两本书之一来入门 js）</p>
<h3 id="正片"><a href="#正片" class="headerlink" title="正片"></a>正片</h3><p>上面无论是代码还是文字，都不是我这种智商能够轻易理解的。我的大脑能够正常处理的代码应该长这样：<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">function</span> <span class="title function_">add</span>(<span class="params">x, y</span>)&#123;</span><br><span class="line">    <span class="keyword">return</span> x + y;</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="title function_">add</span>(<span class="number">1</span>, <span class="number">2</span>)   <span class="comment">// =&gt; 3</span></span><br></pre></td></tr></table></figure><br>在学过一些函数式编程入门知识后，勉强可以接受这样：<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">function</span> <span class="title function_">add</span>(<span class="params">x</span>)&#123;</span><br><span class="line">    <span class="keyword">return</span> <span class="keyword">function</span> <span class="title function_">addx</span>(<span class="params">y</span>)&#123;</span><br><span class="line">        <span class="keyword">return</span> x + y;</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="title function_">add</span>(<span class="number">1</span>)(<span class="number">2</span>)   <span class="comment">// =&gt; 3</span></span><br></pre></td></tr></table></figure><br>以上将原本接受两个参数的函数转换为只接受单个参数的函数，这个过程又称为 <a href="https://zh.wikipedia.org/wiki/%E6%9F%AF%E9%87%8C%E5%8C%96">柯里化</a>。柯里化后的 add 函数返回值是另一个函数。一些程序员可能并不习惯这种用法，因为在某些编程语言中函数只能用来操作数据，不能操作函数。而在 js 中函数也是数据的一种，用面向对象程序员熟悉的话来说就是，函数是数据的子类型，它们之间满足<a href="https://en.wikipedia.org/wiki/Liskov_substitution_principle">里氏替换原则</a>。也就是说在 js 中， 函数实现了数据拥有的所有行为，可以去任何数据能够去的地方。<br>现在不妨来思考一下，我们应该用什么样的数据结构来表达函数这种数据类型？最简单粗暴的方法莫过于直接使用函数的源码字符串，早期的 lisp 语言就是这么做的。但是这种做法有一个致命的缺陷，考虑上面调用<code>add(1)</code>后返回的这个函数：<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">function</span> <span class="title function_">addx</span>(<span class="params">y</span>)&#123;</span><br><span class="line">    <span class="keyword">return</span> x + y;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br>它的字符串表示形式是 <code>&quot;function addx(y)&#123;return x + y;&#125;&quot;</code>，当我们试图调用它时会发现没办法解释其中的<a href="http://blog.staynoob.cn/post/2017/03/lambda-calculus-introduction/#2-绑定变量与自由变量">自由变量</a> x，一个可能的解决方案是采取就近原则，在该函数的调用栈中去寻找 x 的值，于是产生了如下让人匪夷所思的执行结果：<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="title function_">add</span>(<span class="number">1</span>)(<span class="number">2</span>)         <span class="comment">// =&gt; NaN   因为执行环境没有定义x, 所以得到undefined + 2;</span></span><br><span class="line"><span class="keyword">const</span> x = <span class="number">100</span>;</span><br><span class="line"><span class="title function_">add</span>(<span class="number">1</span>)(<span class="number">2</span>)         <span class="comment">// =&gt; 102   在执行环境找到x，所以得到100 + 2</span></span><br></pre></td></tr></table></figure><br>上面这种解释方案又被称为动态作用域(dynamic scoping)，虽然它的确实现了将函数作为数据传递，但是也直接导致了函数的行为无法预测。因此，现代编程语言普遍采用的是另一种称为词法作用域（lexical scoping）的解释方案。这种方案强调的是，应该使用函数定义时的作用域来解释函数中的自由变量，这样才能让函数的行为与定义时预期的行为保持一致。为了实现这一点，很容易想到应该把用来保存函数的数据结构改成如下形式：<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">&#123;</span><br><span class="line">    <span class="attr">func</span>: <span class="string">&quot;function addx(y)&#123;return x + y;&#125;&quot;</span>,</span><br><span class="line">    <span class="attr">scope</span>: &#123;</span><br><span class="line">        <span class="attr">x</span>: <span class="number">1</span>,</span><br><span class="line">    &#125;,</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br>而这种既包含函数的代码逻辑，又包含函数定义时作用域的<strong>数据结构</strong>就是闭包。现在当执行函数遇到自由变量时，直接在闭包中查找定义时该变量的值就可以了。<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">const</span> addx = <span class="title function_">add</span>(<span class="number">1</span>);</span><br><span class="line"><span class="title function_">addx</span>(<span class="number">2</span>)         <span class="comment">// =&gt; 3</span></span><br><span class="line"><span class="keyword">const</span> x = <span class="number">100</span>;</span><br><span class="line"><span class="title function_">addx</span>(<span class="number">2</span>)         <span class="comment">// =&gt; 3</span></span><br></pre></td></tr></table></figure><br>至此，闭包的概念也就解释完了。</p>
<h3 id="Function-prototype-bind"><a href="#Function-prototype-bind" class="headerlink" title="Function.prototype.bind"></a>Function.prototype.bind</h3><p>最后再提一个 js 新手可能不知道的小技巧，使用函数的 bind 方法，可以将函数的绑定变量转换为自由变量，同时将该变量加入闭包作用域。继续拿上面的例子来说，原函数是：<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">function</span> <span class="title function_">add</span>(<span class="params">x, y</span>)&#123;</span><br><span class="line">    <span class="keyword">return</span> x + y;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br>可以通过<code>const addx = add.bind(null, 1)</code>来得到一个跟上面调用<code>add(1)</code>返回的函数等价的函数。</p>
]]></content>
      
        <categories>
            
            <category> Frontend </category>
            
        </categories>
        
        
        <tags>
            
            <tag> Javascript </tag>
            
        </tags>
        
    </entry>
    
    <entry>
      <title><![CDATA[B-tree数据结构]]></title>
      <url>https://blog.staynoob.cn/post/2017/06/B-tree%25E6%2595%25B0%25E6%258D%25AE%25E7%25BB%2593%25E6%259E%2584/</url>
      <content type="html"><![CDATA[<blockquote>
<p>几乎所有文件系统索引，数据库索引，都需要用到B-tree或其变种。因此对于码农而言，理解它如何工作还是相当有必要的。然而我最近阅读了很多相关的文章，它们往往一上来就试图跟读者解释<code>branchingFactor/degree, B-1&lt;=keys&lt;2B-1,B&lt;=children&lt;2B blablabla...</code>，在读者还不懂原理时就开始描述实现细节。导致像我一样没学过正统cs课程的野生码农严重怀疑自己的智商。这篇文章试图向跟我一样的小白来描述B-tree（前提是你能理解二分查找）。</p>
</blockquote>
<h3 id="一-B-tree-B-tree-B-tree"><a href="#一-B-tree-B-tree-B-tree" class="headerlink" title="一.B tree,B-tree,B+tree"></a>一.B tree,B-tree,B+tree</h3><p>如果你在google搜索B-tree数据结构，你可能会找到以上三个关键字相关的文章，这一度让我觉得它们是三种不同的数据结构(既然有BPlusTree，很容易让人想到还有BMinusTree，再加上一些不负责任的翻译版算法书竟然丧心病狂的使用<strong> B- tree </strong>(注意空格)来标识B-tree)。事实上这里的<strong> - </strong>只是连字符，B-tree与B tree是同一回事。而B+tree则是它们的一个演化版。当然你也许还会发现一些文章中的B-tree和另一些文章中的B tree有着不同的实现。请依然不要被误导，我读了大概20来篇B-tree相关的文章，至今没有发现两个完全相同的实现。。。</p>
<span id="more"></span>
<h3 id="二-Trees"><a href="#二-Trees" class="headerlink" title="二.Trees"></a>二.Trees</h3><h4 id="BinarySearchTree"><a href="#BinarySearchTree" class="headerlink" title="BinarySearchTree"></a>BinarySearchTree</h4><p>下图展示的是一颗完全平衡的二叉树，每个节点大于其左子节点，小于其右子节点。当需要查找每个键时，先将其与根节点比较，如果小于根节点则继续查找根节点的左子树，否则查找右子树。如果你对该过程还不熟悉，建议先花一些时间看一些入门的算法教程。在理想情况下（如图所示完美平衡的情况下），一颗含有N个节点的二叉树，其高度正好为 $\log (N+1)$。这意味着一次查找最多只会经过 $\log (N+1)$ 个节点。举例来说如果一颗平衡二叉树包含1000个元素，那么树的高度为10，一次查找操作最多经过10个节点。<br><img src="/img/content/B-tree/bst.png" alt="bst"></p>
<h4 id="2-3tree"><a href="#2-3tree" class="headerlink" title="2-3tree"></a>2-3tree</h4><p>接下来，人们发现在每个节点中添加一个key，该算法同样高效。如下图所示，一个节点可以有两个key和三条链接，人们亲切的称其为2-3树。2-3树的查找算法与二叉树如出一辙，但由于每个节点可以多存储1个key，进一步降低了树的高度（降低树高并不是2-3树的主要目的，主要目的在于自平衡）。很容易可以得出结论，一棵含有N个节点的2-3树其高度在 $\log N$ (2-3数允许包含只有一个key的节点)和 $\log _3N$ 之间。也就是说一棵含有10亿个key的2-3树，其高度在19到30之间。<br><img src="/img/content/B-tree/2-3tree.png" alt="2-3tree"></p>
<h4 id="RedBlackTree-乱入"><a href="#RedBlackTree-乱入" class="headerlink" title="RedBlackTree(乱入)"></a>RedBlackTree(乱入)</h4><p>2-3树对比二叉树的主要优势在于，为了在插入数据时维护树的平衡，你不必再深陷于节点间的左旋转，右旋转无法自拔。2-3树只有根节点分裂的情况下会增加树的高度（这篇文章只谈搜索，对插入与删除感兴趣可以查看文章末尾的链接）。而它的一个缺点在于，在实现中要抽象这种既可能有一个key，又可能有两个key的节点有些繁琐，于是出现了下图所示的数据结构。所有的节点再度回归到只有一个key，并通过两个节点来组成一个含有两个key的节点。为了区分它们与普通二叉树节点的不同，将它们之间的链接标记为红色。江湖人称红黑树。下图所示的红黑树完全等价于上面的2-3树。<br><img src="/img/content/B-tree/rbtree.png" alt="rbtree"></p>
<h4 id="B-tree"><a href="#B-tree" class="headerlink" title="B-tree"></a>B-tree</h4><p>既然我们可以在每个节点中添加一个key，那为什么不干脆添加多个key？于是就出现了类似下图的数据结构，也就是本文的主角B-tree。在B-tree中一个节点可以保存多个key，但具体可以保存多少个key必须有一个参数来加以控制，如果一个节点可以保存无限个key，那它就跟链表没什么区别了，太少又达不到降低树高的效果。接下来，我们需要为这个参数取一个高大上的名字，好让它可以糊弄那些初学者，比如说branchingFactor或者degree就都还不错，在本文中将使用branchingFactor(简写为B)标识该参数，并约定每个节点的key数量大于等于B-1，小于2B-1。这样一来，我们就可以说一棵2-3树是一棵B等于2的B-tree了。<br><img src="/img/content/B-tree/b-tree.png" alt="b-tree"></p>
<h4 id="小结"><a href="#小结" class="headerlink" title="小结"></a>小结</h4><p>上面的树结构各式各样，但查找元素的算法都大同小异，无非是利用每次比较的信息量来确定下一步要查找的分支。用我这个野生程序员的话来总结就是：2-3tree是B等于2的B-tree，红黑树是2-3树的一个特殊实现，同时还是一颗平衡二叉树。至于 rotate,flip,split 什么的，让它随风去吧。</p>
<h3 id="三-why-B-tree"><a href="#三-why-B-tree" class="headerlink" title="三.why B-tree?"></a>三.why B-tree?</h3><p>由于每个节点可以保存多个key，B-tree的高度通常非常低，当B等于500时，一颗含有625亿个key的B-tree高度不会超过4。然而这并不意味B-tree拥有比平衡二叉树更优的查询复杂度。假设一颗含有N个元素的B-tree每个节点正好保存了B个key，那么一次查询最多经过 $\log _BN$ 个节点，但在每个节点中需进行一次二分查找确定下一步选择哪个节点，因此B-tree查找算法实际的时间复杂度为 $O(\log _BN \times \log B)$，熟悉对数性质的同学能够看出来该值就等于 $O(\log N)$。既然如此，为什么还需要B-tree?。<br>上面的计算模型假设对存储器的每一次读取操作成本是相等的。但实际情况往往并非如此。最简单的例子就是，一次磁盘读取操作的开销要比内存读取大好几个数量级。为了尽量减少磁盘IO的次数，往往每次读取磁盘会额外预读一部分数据到内存中，预读数据以页（又称为<a href="https://zh.wikipedia.org/wiki/%E7%B0%87">簇</a>，NTFS文件系统中默认簇大小为4096字节）为单位。也就是说当你需要在磁盘中读取1个字节的数据，计算机会在磁盘中找到该数据，并以其为起始位置一次性读取多页数据载入内存备用。这样一来如果一个树节点所占空间正好为一页大小，就能保证访问该节点所有key值只会发生一次磁盘读取。按照该理论，如果一页能够容纳1000个key节点（B=500），那么在容纳数百亿key值的B-tree中，最多3次磁盘读取就能完成一次查询操作（根节点一般常驻内存），而一般的二叉树显然无法保证这一点。</p>
<h3 id="四-why-B-tree"><a href="#四-why-B-tree" class="headerlink" title="四.why B+tree?"></a>四.why B+tree?</h3><p>为了更好的利用B-tree的优势，必须确保一个树节点所占空间小于文件系统页。也就是说假设一个B-tree节点中的key数量为n，必须保证<strong> pageSize &gt;= n(keySize + dataSize + pointSize) + pointSize </strong>。<br><img src="/img/content/B-tree/b-tree-with-data.png" alt="b-tree-with-data"></p>
<p>上图展示了一个典型的B-tree节点，其中最大的问题在于，如果data所占空间太大，那么我们不得不选择一个较小的B值，而B值过小又无法保证合理的树高。更加麻烦的是，在实现B-tree时，我们往往并不知道data有多大。考虑到上述这些困境，B+tree应运而生。下图展示了典型的B+tree节点，其中最大的变化在于，内部节点不再保存数据，因此每个内部节点得到了更多的空间来存储key值，进一步发挥了B-tree的优势。同时外部节点的实现也有了更多的想象空间，它可以是硬盘中的一页数据，也可以是网络中的某台计算机。如果是用来实现数据库索引，通常还会在外部节点之间添加指向下一节点的指针，这样可以更好的支持范围查询操作。<br><img src="/img/content/B-tree/b+tree.png" alt="b+tree"></p>
<h3 id="五-总结"><a href="#五-总结" class="headerlink" title="五.总结"></a>五.总结</h3><p>以上内容仅仅介绍了B-tree的基本结构以及它要解决的问题，不过我认为对于入门玩家来说，这就足够了。想要更加深入理解或是自己实现B-tree的读者可以查看下面的链接。最后附上我的 <a href="https://github.com/noob9527/algorithm-ts/tree/master/src/structure/dictionary/BPlusTree">B+tree(typescript)</a> 实现以供参考。</p>
<ul>
<li><a href="https://www.youtube.com/watch?v=TOb1tuEZ2X4&amp;t=1124s">R2. 2-3 Trees and B-Trees(推荐观看，虽然印度小哥的口音实在让我蛋碎)</a></li>
<li><a href="http://blog.codinglabs.org/articles/theory-of-mysql-index.html">MySQL索引背后的数据结构及算法原理</a></li>
</ul>
]]></content>
      
        <categories>
            
            <category> Algorithm </category>
            
        </categories>
        
        
    </entry>
    
    <entry>
      <title><![CDATA[lambda calculus:Y-combinator]]></title>
      <url>https://blog.staynoob.cn/post/2017/03/lambda-calculus-Y-combinator/</url>
      <content type="html"><![CDATA[<blockquote>
<p>上篇文章的末尾提到了一个问题，如何使用λ演算实现递归函数？其中最出名的解决方案是由数学家<a href="https://en.wikipedia.org/wiki/Haskell_Curry">Haskell B. Curry</a>发现的一个被称为Y-combinator的函数。</p>
</blockquote>
<h3 id="一-推导过程"><a href="#一-推导过程" class="headerlink" title="一.推导过程"></a>一.推导过程</h3><p>Y-combinator简单来说就是一个输入函数，返回该函数递归版本的函数。关于它的推导我读了很多文章，以下是我总结的一个<em>个人认为</em>比较好理解的版本。（本文代码，可以在<a href="https://github.com/noob9527/y-combinator-js/blob/master/demo.js">y-combinator-js</a>仓库中下载）</p>
<h4 id="Step1"><a href="#Step1" class="headerlink" title="Step1"></a>Step1</h4><p>首先回到大家学习递归函数的起点，阶乘函数，以下是js版本：<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">const</span> <span class="title class_">FACT10</span> = <span class="number">3628800</span>;</span><br><span class="line"><span class="keyword">function</span> <span class="title function_">factorial</span>(<span class="params">n</span>) &#123;</span><br><span class="line">    <span class="keyword">return</span> n === <span class="number">0</span> ? <span class="number">1</span> : n * <span class="title function_">factorial</span>(n - <span class="number">1</span>);</span><br><span class="line">&#125;</span><br><span class="line"><span class="title function_">factorial</span>(<span class="number">10</span>).<span class="property">should</span>.<span class="title function_">equal</span>(<span class="title class_">FACT10</span>);</span><br></pre></td></tr></table></figure><br>现在我们的问题是如何将其转换为合法的λ表达式，换句话说如何将其转换为匿名函数？唯一的可能就是把factorial作为参数传入，下面是修改后的版本：<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">//注意下面的anonymous函数名是不必要的，可以直接写成立即执行函数(IIFE)，之所以我没那样写是为了读起来更清晰</span></span><br><span class="line"><span class="comment">//es5</span></span><br><span class="line"><span class="keyword">function</span> <span class="title function_">anonymous1</span>(<span class="params">factorial</span>) &#123;</span><br><span class="line">    <span class="keyword">return</span> <span class="keyword">function</span> (<span class="params">n</span>) &#123;</span><br><span class="line">        <span class="keyword">return</span> n === <span class="number">0</span> ? <span class="number">1</span> : n * <span class="title function_">factorial</span>(n - <span class="number">1</span>);</span><br><span class="line">    &#125;;</span><br><span class="line">&#125;</span><br><span class="line"><span class="comment">//es6</span></span><br><span class="line"><span class="keyword">const</span> <span class="title function_">anonymous2</span> = f =&gt; (<span class="function"><span class="params">n</span> =&gt;</span> n == <span class="number">0</span> ? <span class="number">1</span> : n * <span class="title function_">f</span>(n - <span class="number">1</span>));</span><br><span class="line"><span class="title function_">anonymous1</span>(factorial)(<span class="number">10</span>).<span class="property">should</span>.<span class="title function_">equal</span>(<span class="title class_">FACT10</span>);</span><br><span class="line"><span class="title function_">anonymous2</span>(factorial)(<span class="number">10</span>).<span class="property">should</span>.<span class="title function_">equal</span>(<span class="title class_">FACT10</span>);</span><br></pre></td></tr></table></figure><br>现在已经可以使用λ表达式来描述上面的递归函数了，λ演算版本大概长这样(看不懂不要紧，描述的意思跟上面的anonymous函数是一样的)<br><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">λf.λn.ISZERO n 1 (MULT n (f (PRED n)))</span><br></pre></td></tr></table></figure><br>不过事情并没有这么简单，仔细想想我们刚才干了啥？我们定义了一个阶乘函数，但是这个阶乘函数却需要一个已经存在的阶乘函数作为参数才可以正常工作，它就像这个手电筒。</p>
<div align="center"><br><img src="/img/content/electric-torch.png" alt="electric torch"><br></div>

<p>换句话说，如果给上面的anonymous函数传入一个阶乘函数，它就能返回一个阶乘函数，如果传入的不是阶乘函数，返回的也肯定不是阶乘函数（请叫我达文西...）。</p>
<span id="more"></span>
<h4 id="Step2"><a href="#Step2" class="headerlink" title="Step2"></a>Step2</h4><p>上面的函数没办法满足我们的需求。我们要的函数，必须输入一个不是阶乘函数的函数，返回一个阶乘函数，就像我们要的手电筒必须输入电能（或别的什么“能”），输出光能。总之不能是“输入光能，输出光能”。因此，把上面的版本改一下，这次我不要求输入阶乘函数了，你就给我一个<strong>自己调用自己就能产生阶乘函数</strong>的函数吧。<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">function</span> <span class="title function_">anonymous</span>(<span class="params">whatever</span>) &#123;</span><br><span class="line">    <span class="keyword">return</span> <span class="keyword">function</span> (<span class="params">n</span>) &#123;</span><br><span class="line">        <span class="keyword">return</span> n === <span class="number">0</span> ? <span class="number">1</span> : n * (<span class="title function_">whatever</span>(whatever))(n - <span class="number">1</span>);</span><br><span class="line">    &#125;;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br>经过上面的修改之后，anonymous函数要求的输入是一个自己调用自己能返回阶乘函数的函数，输出是我们要的阶乘函数，可是我们上哪去找自己调用自己能返回阶乘函数的函数呢？这一步需要点脑洞，试想一下，假如我们用anonymous函数自己调用自己会发生什么？你猜的没错，anonymous函数要求的输入就是它自身，调用方式如下：<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="title function_">anonymous</span>(anonymous)(<span class="number">10</span>).<span class="property">should</span>.<span class="title function_">equal</span>(<span class="title class_">FACT10</span>);</span><br></pre></td></tr></table></figure><br>始终记得anonymous函数名不是必须的，上面的调用已经完全可以使用匿名函数表达：<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">(<span class="keyword">function</span> (<span class="params">whatever</span>) &#123;</span><br><span class="line">    <span class="keyword">return</span> <span class="keyword">function</span> (<span class="params">n</span>) &#123;</span><br><span class="line">        <span class="keyword">return</span> n === <span class="number">0</span> ? <span class="number">1</span> : n * (<span class="title function_">whatever</span>(whatever))(n - <span class="number">1</span>);</span><br><span class="line">    &#125;;</span><br><span class="line">&#125;)(<span class="keyword">function</span> (<span class="params">whatever</span>) &#123;</span><br><span class="line">    <span class="keyword">return</span> <span class="keyword">function</span> (<span class="params">n</span>) &#123;</span><br><span class="line">        <span class="keyword">return</span> n === <span class="number">0</span> ? <span class="number">1</span> : n * (<span class="title function_">whatever</span>(whatever))(n - <span class="number">1</span>);</span><br><span class="line">    &#125;;</span><br><span class="line">&#125;)(<span class="number">10</span>).<span class="property">should</span>.<span class="title function_">equal</span>(<span class="title class_">FACT10</span>);</span><br></pre></td></tr></table></figure><br>等价的λ演算语法如下:<br><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">(λf.λn.ISZERO n 1 (MULT n (f f (PRED n)))(λf.λn.ISZERO n 1 (MULT n (f f (PRED n))))</span><br></pre></td></tr></table></figure><br>现在我们要的匿名递归函数已经创建完成，不过有代码洁癖的同学应该没法接受上面的代码，接下来需要尝试提取出重复的逻辑。</p>
<h4 id="Step3"><a href="#Step3" class="headerlink" title="Step3"></a>Step3</h4><p>首先把碍眼的自身调用自身的逻辑封装成一个单独的callSelf函数：<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">function</span> <span class="title function_">anonymous</span>(<span class="params">whatever</span>) &#123;</span><br><span class="line">    <span class="keyword">function</span> <span class="title function_">callSelf</span>(<span class="params">n</span>) &#123;</span><br><span class="line">        <span class="keyword">return</span> <span class="title function_">whatever</span>(whatever)(n);</span><br><span class="line">    &#125;</span><br><span class="line">    <span class="keyword">return</span> <span class="keyword">function</span> (<span class="params">n</span>) &#123;</span><br><span class="line">        <span class="keyword">return</span> n === <span class="number">0</span> ? <span class="number">1</span> : n * <span class="title function_">callSelf</span>(n - <span class="number">1</span>);</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br>之后可以把callSelf函数作为参数传给返回的函数：<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">function</span> <span class="title function_">anonymous</span>(<span class="params">whatever</span>) &#123;</span><br><span class="line">    <span class="keyword">function</span> <span class="title function_">callSelf</span>(<span class="params">n</span>) &#123;</span><br><span class="line">        <span class="keyword">return</span> <span class="title function_">whatever</span>(whatever)(n);</span><br><span class="line">    &#125;;</span><br><span class="line">    <span class="keyword">return</span> (<span class="keyword">function</span> (<span class="params">whatever</span>) &#123;</span><br><span class="line">        <span class="keyword">return</span> <span class="keyword">function</span> (<span class="params">n</span>) &#123;</span><br><span class="line">            <span class="keyword">return</span> n === <span class="number">0</span> ? <span class="number">1</span> : n * <span class="title function_">whatever</span>(n - <span class="number">1</span>);</span><br><span class="line">        &#125;;</span><br><span class="line">    &#125;)(callSelf);</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br>也许你已经发现了，返回的那一坨跟我们在<a href="#Step1">Step1</a>中定义的函数有点像！把它拎出来围观一下：<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">function</span> <span class="title function_">step1_anonymous</span>(<span class="params">factorial</span>) &#123;</span><br><span class="line">    <span class="keyword">return</span> <span class="keyword">function</span> (<span class="params">n</span>) &#123;</span><br><span class="line">        <span class="keyword">return</span> n === <span class="number">0</span> ? <span class="number">1</span> : n * <span class="title function_">factorial</span>(n - <span class="number">1</span>);</span><br><span class="line">    &#125;;</span><br><span class="line">&#125;</span><br><span class="line"><span class="keyword">function</span> <span class="title function_">anonymous</span>(<span class="params">whatever</span>)&#123;</span><br><span class="line">    <span class="keyword">function</span> <span class="title function_">callSelf</span>(<span class="params">n</span>) &#123;</span><br><span class="line">        <span class="keyword">return</span> <span class="title function_">whatever</span>(whatever)(n);</span><br><span class="line">    &#125;;</span><br><span class="line">    <span class="keyword">return</span> <span class="title function_">step1_anonymous</span>(callSelf);</span><br><span class="line">&#125;</span><br><span class="line"><span class="title function_">anonymous</span>(anonymous)(<span class="number">10</span>).<span class="property">should</span>.<span class="title function_">equal</span>(<span class="title class_">FACT10</span>);</span><br></pre></td></tr></table></figure><br>现在代码总算看起来舒服多了，不过还有个问题，总不能每次都像<code>anonymous(anonymous)(10)</code>这样调用阶乘函数吧？既然<code>anonymous(anonymous)</code>会返回我们要的函数，干脆把它封装成一个factorialFactory(工厂函数)，这个函数还可以接受一个参数，顺便把<strong>step1_anonymous</strong>传进去就好了：<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">function</span> <span class="title function_">step1_anonymous</span>(<span class="params">factorial</span>) &#123;</span><br><span class="line">    <span class="keyword">return</span> <span class="keyword">function</span> (<span class="params">n</span>) &#123;</span><br><span class="line">        <span class="keyword">return</span> n === <span class="number">0</span> ? <span class="number">1</span> : n * <span class="title function_">factorial</span>(n - <span class="number">1</span>);</span><br><span class="line">    &#125;;</span><br><span class="line">&#125;</span><br><span class="line"><span class="keyword">function</span> <span class="title function_">factorialFactory</span>(<span class="params">step1_anonymous</span>) &#123;</span><br><span class="line">    <span class="comment">//return anonymous(anonymous);</span></span><br><span class="line">    <span class="keyword">return</span> (<span class="keyword">function</span>(<span class="params">whatever</span>) &#123;</span><br><span class="line">        <span class="keyword">function</span> <span class="title function_">callSelf</span>(<span class="params">n</span>) &#123;</span><br><span class="line">            <span class="keyword">return</span> <span class="title function_">whatever</span>(whatever)(n);</span><br><span class="line">        &#125;;</span><br><span class="line">        <span class="keyword">return</span> <span class="title function_">step1_anonymous</span>(callSelf);</span><br><span class="line">    &#125;)(<span class="keyword">function</span>(<span class="params">whatever</span>) &#123;</span><br><span class="line">        <span class="keyword">function</span> <span class="title function_">callSelf</span>(<span class="params">n</span>) &#123;</span><br><span class="line">            <span class="keyword">return</span> <span class="title function_">whatever</span>(whatever)(n);</span><br><span class="line">        &#125;;</span><br><span class="line">        <span class="keyword">return</span> <span class="title function_">step1_anonymous</span>(callSelf);</span><br><span class="line">    &#125;);</span><br><span class="line">&#125;</span><br><span class="line"><span class="title function_">factorialFactory</span>(step1_anonymous)(<span class="number">10</span>).<span class="property">should</span>.<span class="title function_">equal</span>(<span class="title class_">FACT10</span>);</span><br></pre></td></tr></table></figure><br>最后，将里面用到的函数重写成匿名版本：<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">//全箭头函数版本：const Y = f =&gt; (x =&gt; f(n =&gt; x(x)(n)))(x =&gt; f(n =&gt; x(x)(n)));</span></span><br><span class="line"><span class="comment">//λ演算版本：λf.(λx.f(λn.x x n))(λx.f(λn.x x n))</span></span><br><span class="line"><span class="keyword">const</span> Y = <span class="keyword">function</span> (<span class="params">f</span>) &#123;</span><br><span class="line">    <span class="keyword">return</span> (<span class="keyword">function</span> (<span class="params">x</span>) &#123;</span><br><span class="line">        <span class="keyword">return</span> <span class="title function_">f</span>(<span class="function"><span class="params">n</span> =&gt;</span> <span class="title function_">x</span>(x)(n));</span><br><span class="line">    &#125;)(<span class="keyword">function</span> (<span class="params">x</span>) &#123;</span><br><span class="line">        <span class="keyword">return</span> <span class="title function_">f</span>(<span class="function"><span class="params">n</span> =&gt;</span> <span class="title function_">x</span>(x)(n));</span><br><span class="line">    &#125;);</span><br><span class="line">&#125;;</span><br><span class="line"><span class="title function_">Y</span>(step1_anonymous)(<span class="number">10</span>).<span class="property">should</span>.<span class="title function_">equal</span>(<span class="title class_">FACT10</span>);</span><br></pre></td></tr></table></figure><br>由于计算阶乘的逻辑已经全部提取到step1_anonymous函数中，因此factorialFactory已经不再局限于制造阶乘函数，而是一个输入任意函数，就可以返回输入函数的递归版本的函数。也就是本文的主角Y-combinator，下面使用斐波那契数列计算函数来测试一下。<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">const</span> <span class="variable constant_">FIB</span> = [<span class="number">0</span>, <span class="number">1</span>, <span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>, <span class="number">5</span>, <span class="number">8</span>, <span class="number">13</span>, <span class="number">21</span>, <span class="number">34</span>, <span class="number">55</span>, <span class="number">89</span>, <span class="number">144</span>];</span><br><span class="line"><span class="keyword">const</span> fibonacci = <span class="title function_">Y</span>(<span class="function"><span class="params">fib</span> =&gt;</span> (<span class="function"><span class="params">n</span> =&gt;</span> (n &lt;= <span class="number">2</span> ? <span class="number">1</span> : <span class="title function_">fib</span>(n - <span class="number">1</span>) + <span class="title function_">fib</span>(n - <span class="number">2</span>))));</span><br><span class="line"><span class="title function_">fibonacci</span>(<span class="number">5</span>).<span class="property">should</span>.<span class="title function_">equal</span>(<span class="variable constant_">FIB</span>[<span class="number">5</span>]);</span><br><span class="line"><span class="title function_">fibonacci</span>(<span class="number">10</span>).<span class="property">should</span>.<span class="title function_">equal</span>(<span class="variable constant_">FIB</span>[<span class="number">10</span>]);</span><br></pre></td></tr></table></figure><br>以上就是推导<strong>Y</strong>的全部过程，为了让它容易理解，我只展示了必要的步骤。后面还会谈到一些你应该知道的细节。</p>
<h3 id="二-求值策略-evaluation-strategy"><a href="#二-求值策略-evaluation-strategy" class="headerlink" title="二.求值策略(evaluation strategy)"></a>二.求值策略(evaluation strategy)</h3><p>函数调用的求值策略分为传值调用(Eager Evaluation及早求值)与传名调用(Lazy Evaluation惰性求值)，二者的区别在于参数计算时机的不同。考虑下面这个js函数调用：<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">function</span> <span class="title function_">identity</span>(<span class="params">x</span>)&#123;</span><br><span class="line">	<span class="keyword">return</span> x;</span><br><span class="line">&#125;</span><br><span class="line"><span class="keyword">let</span> a = <span class="number">0</span>;</span><br><span class="line"><span class="title function_">identity</span>(<span class="number">0</span>, a++);</span><br><span class="line"><span class="variable language_">console</span>.<span class="title function_">log</span>(a) <span class="comment">//1</span></span><br></pre></td></tr></table></figure><br>identity函数在调用之前就执行了a++表达式，而事实上函数体内根本没有用到该参数，这代表js函数是传值调用。传名调用指的是只有在需要用到的时候，才计算参数的值。大部分情况下，传值调用与传名调用会得到相同的结果。不过也有一些例外，比如上面推导过程的callSelf函数：<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">function</span> <span class="title function_">anonymous</span>(<span class="params">whatever</span>)&#123;</span><br><span class="line">    <span class="keyword">function</span> <span class="title function_">callSelf</span>(<span class="params">n</span>) &#123;</span><br><span class="line">        <span class="keyword">return</span> <span class="title function_">whatever</span>(whatever)(n);</span><br><span class="line">    &#125;;</span><br><span class="line">    <span class="keyword">return</span> <span class="title function_">step1_anonymous</span>(callSelf);</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br>乍一看会发现它应该可以写成这样：<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">function</span> <span class="title function_">anonymous</span>(<span class="params">whatever</span>)&#123;</span><br><span class="line">    <span class="keyword">return</span> <span class="title function_">step1_anonymous</span>(<span class="title function_">whatever</span>(whatever));</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br>这种写法在传名调用的语言中可以正常工作，但是在像js这样的传值调用语言中不行，因为在step1_anonymous调用之前，就会计算whatever(whatever)的值，然后无限递归导致<strong>stackoverflow</strong>异常。而将<code>whatever(whatever)</code>改写成<code>n=&gt;whatever(whatever)(n)</code>使用的正是在上篇文章中提到的<a href="/post/2017/03/lambda-calculus-introduction/#3-化简规则">η变换(Eta-conversion)</a>。下面分别列出在两种求值策略中的Y：</p>
<ul>
<li>传值调用：<code>λf.(λx.f(λn.x x n))(λx.f(λn.x x n))</code></li>
<li>传名调用：<code>λf.(λx.f(x x))(λx.f(x x))</code></li>
</ul>
<p>目前我只知道haskell采取传名调用的求值策略，其它主流编程语言(java,c,js...)都采取传值调用，关于二者更详细的分析可以参考<a href="http://www.yinwang.org/blog-cn/2013/04/01/lazy-evaluation">这篇文章</a>。</p>
<h3 id="三-函数的不动点-fix-point"><a href="#三-函数的不动点-fix-point" class="headerlink" title="三.函数的不动点(fix point)"></a>三.函数的不动点(fix point)</h3><p>让我们再度回到<a href="/#Step1">Step1</a>中的阶乘函数，这个函数在计算0的阶乘时，不需要调用传入的函数，因此传入任意一个函数，都会返回一个能够正确计算0!的函数，这里先取名叫fact0：<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">function</span> <span class="title function_">anonymous</span>(<span class="params">factorial</span>) &#123;</span><br><span class="line">    <span class="keyword">return</span> <span class="keyword">function</span> (<span class="params">n</span>) &#123;</span><br><span class="line">        <span class="keyword">return</span> n === <span class="number">0</span> ? <span class="number">1</span> : n * <span class="title function_">factorial</span>(n - <span class="number">1</span>);</span><br><span class="line">    &#125;;</span><br><span class="line">&#125;</span><br><span class="line"><span class="keyword">function</span> <span class="title function_">whatever</span>(<span class="params">x</span>) &#123;</span><br><span class="line">    <span class="keyword">throw</span> <span class="keyword">new</span> <span class="title class_">Error</span>(<span class="string">&#x27;Gotcha!&#x27;</span>);</span><br><span class="line">&#125;</span><br><span class="line"><span class="keyword">const</span> fact0 = <span class="title function_">anonymous</span>(whatever);</span><br><span class="line"><span class="title function_">fact0</span>(<span class="number">0</span>).<span class="property">should</span>.<span class="title function_">equal</span>(<span class="number">1</span>);</span><br><span class="line">t.<span class="title function_">throws</span>(<span class="function">() =&gt;</span> <span class="title function_">fact0</span>(<span class="number">1</span>), <span class="string">&#x27;Gotcha!&#x27;</span>);</span><br></pre></td></tr></table></figure><br>上面的代码展示了使用fact0计算0的阶乘，能够得到正确的结果1。但是如果使用它计算1的阶乘我们就露陷了，因为它需要调用传入的函数来计算<strong>n-1(也就是0)</strong>的阶乘。换句话说，如果需要得到能够计算1的阶乘的函数，我们需要传入一个可以正确计算0的阶乘的函数。幸运的是，手头上的fact0就是这个函数，因此把fact0作为参数再次调用anonymous就能得到一个可以计算1的阶乘的函数：<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">const</span> fact1 = <span class="title function_">anonymous</span>(fact0)	<span class="comment">//也就是anonymous(anonymous(whatever));</span></span><br><span class="line"><span class="title function_">fact1</span>(<span class="number">1</span>).<span class="property">should</span>.<span class="title function_">equal</span>(<span class="number">1</span>);</span><br><span class="line">t.<span class="title function_">throws</span>(<span class="function">() =&gt;</span> <span class="title function_">fact1</span>(<span class="number">2</span>), <span class="string">&#x27;Gotcha!&#x27;</span>);</span><br></pre></td></tr></table></figure><br>同理，使用fact1作为参数就可以得到能够计算2的阶乘的函数，我们可以重复这个过程直到我们满意为止。<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">const</span> fact0 = <span class="title function_">anonymous</span>(whatever);</span><br><span class="line"><span class="keyword">const</span> fact1 = <span class="title function_">anonymous</span>(<span class="title function_">anonymous</span>(whatever));</span><br><span class="line"><span class="keyword">const</span> fact2 = <span class="title function_">anonymous</span>(<span class="title function_">anonymous</span>(<span class="title function_">anonymous</span>(whatever)));</span><br><span class="line"><span class="keyword">const</span> fact3 = <span class="title function_">anonymous</span>(<span class="title function_">anonymous</span>(<span class="title function_">anonymous</span>(<span class="title function_">anonymous</span>(whatever))));</span><br><span class="line"><span class="comment">// const factn = anonymous(anonymous(anonymous(n...)))</span></span><br><span class="line"></span><br><span class="line"><span class="title function_">fact0</span>(<span class="number">0</span>).<span class="property">should</span>.<span class="title function_">equal</span>(<span class="title function_">factorial</span>(<span class="number">0</span>));</span><br><span class="line">t.<span class="title function_">throws</span>(<span class="function">() =&gt;</span> <span class="title function_">fact0</span>(<span class="number">1</span>), <span class="string">&#x27;Gotcha!&#x27;</span>);</span><br><span class="line"><span class="title function_">fact1</span>(<span class="number">1</span>).<span class="property">should</span>.<span class="title function_">equal</span>(<span class="title function_">factorial</span>(<span class="number">1</span>));</span><br><span class="line">t.<span class="title function_">throws</span>(<span class="function">() =&gt;</span> <span class="title function_">fact1</span>(<span class="number">2</span>), <span class="string">&#x27;Gotcha!&#x27;</span>);</span><br><span class="line"><span class="title function_">fact2</span>(<span class="number">2</span>).<span class="property">should</span>.<span class="title function_">equal</span>(<span class="title function_">factorial</span>(<span class="number">2</span>));</span><br><span class="line">t.<span class="title function_">throws</span>(<span class="function">() =&gt;</span> <span class="title function_">fact2</span>(<span class="number">3</span>), <span class="string">&#x27;Gotcha!&#x27;</span>);</span><br><span class="line"><span class="title function_">fact3</span>(<span class="number">3</span>).<span class="property">should</span>.<span class="title function_">equal</span>(<span class="title function_">factorial</span>(<span class="number">3</span>));</span><br><span class="line">t.<span class="title function_">throws</span>(<span class="function">() =&gt;</span> <span class="title function_">fact3</span>(<span class="number">4</span>), <span class="string">&#x27;Gotcha!&#x27;</span>);</span><br><span class="line"><span class="comment">// factn(n).should.equal(factorial(n));</span></span><br></pre></td></tr></table></figure><br>现在假设我们要计算n的阶乘，我们有两种选择，一种是将上面的过程重复n次，得到一个能够计算n的阶乘的函数。另一种是找一个函数<strong> fix </strong>，使得<strong> fix = anonymous(fix) </strong>，这样就不再需要重复n次了，因为不管重复多少次，得到的结果都一样。这里的<strong> fix </strong>就称之为函数anonymous的不动点(fix point)。比如说x=0，就是函数f(x)=x^2的不动点，因为0=f(0)=f(f(0))...。<br>根据上面的分析，我们可以定义一个函数<strong> Y </strong>，<strong> Y </strong>接收一个函数作为参数，返回这个函数的不动点。即有<strong> Y(f) = fix = f(fix) = f(Y(f)) </strong>，根据这条规则，很容易使用js递归函数来定义Y：<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">function</span> <span class="title function_">Y</span>(<span class="params">f</span>) &#123;</span><br><span class="line">    <span class="keyword">return</span> <span class="title function_">f</span>(<span class="title function_">Y</span>(f));</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br>现在可以尝试把anonymous函数传进去，看看它是否跟我们想的一样有效。由于js的急性求值策略，你会发现像<code>Y(anonymous)</code>这样的调用，会导致<strong>stackoverflow</strong>异常。所幸的是经过之前的学习，我们已经知道可以使用<strong> η变换 </strong>来避免这个问题。下面改写这个函数：<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">//eta conversation Y(f) = λx.Y(f)(x)</span></span><br><span class="line"><span class="keyword">function</span> <span class="title function_">Y</span>(<span class="params">f</span>) &#123;</span><br><span class="line">    <span class="keyword">return</span> <span class="title function_">f</span>(<span class="function"><span class="params">x</span> =&gt;</span> <span class="title function_">Y</span>(f)(x));</span><br><span class="line">&#125;</span><br><span class="line"><span class="title function_">Y</span>(anonymous)(<span class="number">10</span>).<span class="property">should</span>.<span class="title function_">equal</span>(<span class="title class_">FACT10</span>);</span><br></pre></td></tr></table></figure><br>上面的Y函数作用等价于Y-combinator，满足Y函数（即<strong> Y(f) = f(Y(f)) </strong>）定义的<a href="https://wiki.haskell.org/Combinator">combinator</a>被称为不动点组合子(fixed-point combinator)，而Y-combinator只是其中之一。</p>
<h3 id="四-总结"><a href="#四-总结" class="headerlink" title="四.总结"></a>四.总结</h3><p>需要注意的是，YC并不能降低算法的复杂度，因此不要尝试在生产环境中使用它，除非你知道自己在做什么。。。</p>
<p>参考链接：</p>
<ul>
<li><a href="https://yinwang0.wordpress.com/2012/04/09/reinvent-y/">How to reinvent the Y combinator</a></li>
<li><a href="http://shellfly.org/blog/2015/01/07/yi-the-y-combinator-slight-return/">(译) The Y combinator (Slight Return)</a></li>
<li><a href="https://en.wikipedia.org/wiki/Fixed-point_combinator">Fixed-point combinator</a></li>
</ul>
]]></content>
      
        <categories>
            
            <category> Functional Programming </category>
            
        </categories>
        
        
        <tags>
            
            <tag> Lambda Calculus </tag>
            
        </tags>
        
    </entry>
    
    <entry>
      <title><![CDATA[lambda calculus:Introduction]]></title>
      <url>https://blog.staynoob.cn/post/2017/03/lambda-calculus-Introduction/</url>
      <content type="html"><![CDATA[<blockquote>
<p>λ演算是函数式编程范式的理论基础，然而我读了两本关于fp的书（《java8函数式编程》，《javascript函数式编程》）都没有提到它，因此自己在网上看了一些资料，并写了这篇作为学习心得。</p>
</blockquote>
<h3 id="一-λ演算-lambda-calculus简称LC-与图灵机"><a href="#一-λ演算-lambda-calculus简称LC-与图灵机" class="headerlink" title="一.λ演算(lambda calculus简称LC)与图灵机"></a>一.λ演算(lambda calculus简称LC)与图灵机</h3><p>首先，λ演算的提出者<a href="https://zh.wikipedia.org/wiki/%E9%98%BF%E9%9A%86%E4%BD%90%C2%B7%E9%82%B1%E5%A5%87">丘奇(Alonzo Church)</a>是普林斯顿大学的教授。1935年，丘奇发表论文使用λ演算证明基本数论中存在不可解决的问题。1936年4月，丘奇指出自己的那篇论文可以推论出著名的<a href="https://zh.wikipedia.org/wiki/%E6%B1%BA%E5%AE%9A%E6%80%A7%E5%95%8F%E9%A1%8C">判定性问题(Hilbert Decision-problem)</a>是不可解决的。1936年5月，图灵发表论文使用他自己假想的计算机器(后被称为图灵机)证明了同一个问题。当时丘奇几乎是世界上唯一能够验证这篇论文正确性的人。因此丘奇将图灵收入门下读博士。之后他们共同提出了<a href="https://zh.wikipedia.org/wiki/%E9%82%B1%E5%A5%87%EF%BC%8D%E5%9B%BE%E7%81%B5%E8%AE%BA%E9%A2%98">邱奇-图灵论题(Church–Turing thesis)</a>，该论题的核心思想是</p>
<blockquote>
<p>如果某个算法是<a href="https://zh.wikipedia.org/wiki/%E5%8F%AF%E8%AE%A1%E7%AE%97%E6%80%A7%E7%90%86%E8%AE%BA">可计算的(Computability)</a>，那这个算法同样可以被图灵机，以及λ演算所实现，图灵机与λ演算是等价的。</p>
</blockquote>
<span id="more"></span>
<p>这意味着假如你现在要自己设计一套编程语言，如果你的语言能做到图灵机或λ演算同样的事情，那么你的语言就可以解决所有的可计算问题。这时，你可以声称它是一门<a href="https://zh.wikipedia.org/wiki/%E5%9C%96%E9%9D%88%E5%AE%8C%E5%82%99%E6%80%A7">图灵完备</a>的编程语言。下面来看看图灵机与λ演算能做什么事情</p>
<ul>
<li><p>图灵机的基本思想大概是这样：</p>
<blockquote>
<ol>
<li>一条无限长的纸带TAPE。纸带被划分为一个接一个的小格子，每个格子上包含一个来自有限字母表的符号，字母表中有一个特殊的符号表示空白。纸带上的格子从左到右依次被编号为0, 1, 2, ...，纸带的右端可以无限伸展。</li>
<li>一个读写头HEAD。该读写头可以在纸带上左右移动，它能读出当前所指的格子上的符号，并能改变当前格子上的符号。</li>
<li>一套控制规则TABLE。它根据当前机器所处的状态以及当前读写头所指的格子上的符号来确定读写头下一步的动作，并改变状态寄存器的值，令机器进入一个新的状态。</li>
<li>一个状态寄存器。它用来保存图灵机当前所处的状态。图灵机的所有可能状态的数目是有限的，并且有一个特殊的状态。称为停机状态。</li>
</ol>
</blockquote>
<p>  这样一台机器就能实现目前人类已知的任何可行算法，以c为代表的命令式编程就是以这种假想模型作为理论基础。</p>
</li>
<li><p>λ演算的基本思想大概是这样：</p>
<blockquote>
<ol>
<li>可以定义函数(function abstraction)</li>
<li>可以调用函数(function application)</li>
</ol>
</blockquote>
<p>  一门语言只要能做到以上两点，就可以实现任何可行算法，以lisp为代表的函数式编程以λ演算作为理论基础。</p>
</li>
</ul>
<p>可以看到λ演算的概念要比图灵机更加简洁明了，顺便再对比一下c与lisp，以下引用<a href="https://book.douban.com/subject/6021440/">《黑客与画家》</a>中的描述</p>
<blockquote>
<p>如果使用Lisp语言，能让程序变得多短？以Lisp和C的比较为例，我听到的大多数说法是C代码的长度是Lisp的7倍到10倍。但是最近，New Architect杂志上有一篇介绍ITA软件公司的文章，里面说&quot;一行Lisp代码相当于20行C代码&quot;，因为此文都是引用ITA总裁的话，所以我想这个数字来自ITA的编程实践。 如果真是这样，那么我们可以相信这句话。ITA的软件，不仅使用Lisp语言，还同时大量使用C和C++，所以这是他们的经验谈。</p>
</blockquote>
<p>最后说说我自己对丘奇和图灵两位伟人的一些感想。两个同样高智商的天才，如果其中一个有情商缺陷，那么那个有情商缺陷的人智商总是会被莫名其妙的放大，这也许就是为什么影视剧作品中总是要以低情商来塑造一个天才（参照福尔摩斯与谢尔顿·库珀）再加上如果某个人的一生以悲剧结尾，那么人们将进一步放大他所做的贡献。关于丘奇与图灵的更多内容可以阅读<a href="http://www.yinwang.org/blog-cn/2015/10/18/turing">图灵的光环</a>。</p>
<h3 id="二-λ演算的语法"><a href="#二-λ演算的语法" class="headerlink" title="二.λ演算的语法"></a>二.λ演算的语法</h3><h4 id="1-λ表达式"><a href="#1-λ表达式" class="headerlink" title="1.λ表达式"></a>1.λ表达式</h4><p>λ表达式只遵循以下三条规则：</p>
<ol>
<li>变量，比如<code>x</code>就是合法的λ表达式</li>
<li>如果y是λ表达式，x是变量，则<code>λx.y</code>是合法的λ表达式（又称为函数声明或lambda abstraction），它代表输入x返回y的匿名函数。等价的js(本文均指<a href="http://www.ecma-international.org/ecma-262/6.0/">ecmascript6</a>)写法是<code>x=&gt;y</code>。<code>λx.x+y</code>也是合法的，它表示一个函数输入x，返回x与未知的y的和。</li>
<li>如果t和s都是λ表达式，则<code>(t s)</code>也是λ表达式（又称为函数应用或application），它代表使用参数s调用函数t，等价的js写法是<code>t(s)</code>。</li>
</ol>
<p>所有合法的λ表达式都是通过重复这三条规则得到的，不过为了保持表达式的整洁，大家一般遵循以下惯例（暂时看不明白也没关系）:</p>
<ol>
<li>最外层的括号可以省略，<code>(M N)</code>可以写成<code>M N</code>。</li>
<li>函数应用左聚合，<code>(M N) P</code>可以写成<code>M N P</code>。</li>
<li>函数定义时，函数体尽可能向右扩展，<code>λx.M N</code>应该解释为<code>λx.(M N)</code>而不是<code>(λx.M) N</code>。</li>
<li>函数定义序列可以被合并表达，<code>λx.λy.λz.N</code>可以被缩写为<code>λxyz.N</code></li>
</ol>
<h4 id="2-绑定变量与自由变量"><a href="#2-绑定变量与自由变量" class="headerlink" title="2.绑定变量与自由变量"></a>2.绑定变量与自由变量</h4><p>如果一个变量是一个λ表达式的参数，则称该变量绑定到该λ上，比如说<code>λx.x+y</code>中，x是绑定变量，y则称为自由变量。更详细的规则如下：</p>
<ul>
<li>λ表达式<code>x</code>中的自由变量就是x</li>
<li>λ表达式<code>λx.t</code>中的自由变量，是t中的自由变量但不包括x，结合第一条规则来看，<code>λx.x</code>的自由变量为空</li>
<li>λ表达式<code>ts</code>中的自由变量，是t中的自由变量与s中的自由变量的并集，结合前两条规则，<code>λx.x x</code>的自由变量是x与空集的并集，即x。</li>
</ul>
<h4 id="3-化简规则"><a href="#3-化简规则" class="headerlink" title="3.化简规则"></a>3.化简规则</h4><p>下面是化简规则的“官方描述”(如果在阅读时出现恶心，头晕等不良反应，可以切换到最后的草根版本)：</p>
<ul>
<li>α变换(Alpha conversion)<br>  α变换简单理解就是λ表达式中的绑定变量可以替换变量名，例如<code>λx.x</code>与<code>λy.y</code>是α等价(Alpha equivalence)的，将<code>λx.x</code>替换成<code>λy.y</code>就称为α变换。这条规则虽然简单，但仍然要小心一些陷阱。考虑下面这个例子，对<code>λx.λx.x</code>进行α变换可以得到<code>λy.λx.x</code>但是不能得到<code>λy.λx.y</code>。其次，当α变换会导致变量被不同的函数绑定时，不允许进行变换，比如<code>λx.λy.x</code>就不能被替换成<code>λy.λy.y</code>。</li>
<li><p>β归约(Beta reduction)<br>  在理解β归约之前，我们先来定义一个变量替换(substitution)操作符，假设M,N是任意λ表达式，x,y是变量，M[x:=N]表示将M中的所有自由变量x替换成表达式N，下面是一些例子：</p>
<ul>
<li>x[x :=N]=N</li>
<li>y[x := N]=y, if x ≠ y</li>
<li>(M1 M2)[x := N] = (M1[x := N]) (M2[x := N])</li>
<li>(λx.M)[x := N]  = λx.M</li>
<li>(λy.M)[x := N]  = λy.(M[x := N]), if x ≠ y, provided y ∉ FV(N)</li>
</ul>
<p>这项操作又称之为Capture-avoiding substitution，因为它必须确保变量替换后不能成为一个绑定变量，比如说<code>(λx.y)[y := x]=λx.x</code>就是不正确的，必须先对<code>λx.y</code>进行α变换，改变绑定变量名得到<code>λz.y</code>，之后再进行[y := x]操作得到<code>λz.x</code>。了解了变量替换后，β归约可以被简单的定义成：</p>
<blockquote>
<p>((λV.E) E&#39;) = E [V:=E&#39;]</p>
</blockquote>
</li>
<li><p>η变换(Eta-conversion)<br>η变换指的是当且仅当两个函数对所有的输入，返回同样的输出时，两个函数是相等的，这意味这x只要不是f中的自由变量，f就可以转换成<code>λx.f x</code>，一些在急性求值(eager evalution)环境下无法正常调用的函数，需要通过这条规则来进行转换，这条规则在后面学习<strong>Y-combinator</strong>的时候会用到，这里大致有个印象就可以了。</p>
</li>
</ul>
<p>草根版本：</p>
<ul>
<li>α变换<br>  函数在不引发变量名冲突的情况下可以修改形参变量名，即<code>x=&gt;x+y</code>等价于<code>z=&gt;z+y</code></li>
<li>β归约<br>  函数可以将实参代入函数体，即<code>(x=&gt;(y=&gt;x+y))(z)</code>等价于<code>y=&gt;z+y</code>，但是如果是<code>(x=&gt;(y=&gt;x+y))(y)</code>则需要先进行α变换得到<code>(x=&gt;(z=&gt;x+z))(y)</code>，再进行β归约得到<code>z=&gt;y+z</code>，而不是直接代入得到<code>y=&gt;y+y</code></li>
<li>η变换<br>  如果f是一个函数，那么<code>f</code>等价于<code>x=&gt;f(x)</code>，因为对于任意变量<code>v</code>，<code>f(v)</code>总是等于<code>(x=&gt;f(x))(v)</code></li>
</ul>
<h3 id="三-简单的运算规则"><a href="#三-简单的运算规则" class="headerlink" title="三.简单的运算规则"></a>三.简单的运算规则</h3><h4 id="1-逻辑运算"><a href="#1-逻辑运算" class="headerlink" title="1.逻辑运算"></a>1.逻辑运算</h4><p>在进行逻辑运算之前，需要先定义什么是真，什么是假。下面的定义称为丘奇布尔值(Church booleans)</p>
<ul>
<li>TRUE = <code>λx.λy.x</code> 对应的js代码是<code>x=&gt;(y=&gt;x)</code>，柯里化之前是<code>(x,y)=&gt;x</code>（等价于丘奇数0）</li>
<li>FALSE = <code>λx.λy.y</code> 对应的js代码是<code>x=&gt;(y=&gt;y)</code>，柯里化之前是<code>(x,y)=&gt;y</code></li>
</ul>
<p>从柯里化之前的js代码来看，丘奇布尔值就是一个接受两个参数的函数，如果为真则返回第一个参数，否则返回第二个(因为合法的λ表达式只接收一个参数，所以才写成了这种形式<code>λx.λy.x</code>，如果你还记得上面介绍的λ表达式的惯例，那么你应该知道<code>λx.λy.x</code>可以简写成<code>λxy.x</code>)。有了布尔值，下面就可以在它的基础上定义出逻辑运算：</p>
<ul>
<li>AND = <code>λp.λq.p q p</code></li>
<li>OR = <code>λp.λq.p p q</code></li>
<li>NOT = <code>λp.p FALSE TRUE</code></li>
<li>IFTHENELSE = <code>λp.λa.λb.p a b</code></li>
</ul>
<p>下面简单验证<strong> AND </strong>表达式的正确性，尝试化简<code>AND TRUE FALSE</code></p>
<ol>
<li>第一步得到 <code>(λp.(λq.((p q) p))) TRUE FALSE</code></li>
<li>第一次β归约得到<code>(λq.((TRUE q) TRUE) FALSE</code></li>
<li>第二次β归约得到<code>TRUE FALSE TRUE</code>即<code>((λx.λy.x) FALSE) TRUE</code></li>
<li>第三次β归约得到<code>(λy.FALSE) TRUE</code>即<code>FALSE</code></li>
</ol>
<h4 id="2-算术运算"><a href="#2-算术运算" class="headerlink" title="2.算术运算"></a>2.算术运算</h4><p>λ演算中有多种方法可以用来定义自然数，如下定义又称为丘奇数(Church numerals)，是使用最广泛的定义方式</p>
<ul>
<li>0 = <code>λf.λx.x</code> 对应的js代码是<code>f=&gt;(x=&gt;x)</code>，柯里化之前是<code>(f,x)=&gt;x</code></li>
<li>1 = <code>λf.λx.f x</code> 对应的js代码是<code>f=&gt;(x=&gt;f(x))</code>，柯里化之前是<code>(f,x)=&gt;f(x)</code></li>
<li>2 = <code>λf.λx.f (f x)</code> 对应的js代码是<code>f=&gt;(x=&gt;f(f(x))</code>，柯里化之前是<code>(f,x)=&gt;f(f(x))</code></li>
<li>3 = <code>λf.λx.f (f (f x))</code> 对应的js代码是<code>f=&gt;(x=&gt;f(f(f(x)))</code>，柯里化之前是<code>(f,x)=&gt;f(f(f(x)))</code></li>
</ul>
<p>同样从柯里化之前的js代码来看，丘奇数是一个接受两个参数的函数，对于任意数字n，它把第一个参数应用到第二个参数上n次。根据这样的思路，可以轻松实现自增函数(输入n，返回n+1的函数)<br><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">SUCC = λn.λf.λx.f (n f x)</span><br></pre></td></tr></table></figure><br>上面的实现思路是输入丘奇数n，在n的基础上再调用一次函数f，就能得到n+1。可以将上面的丘奇数代入该函数进行验算（跟上面的验算步骤一样，不断进行β归约即可）。有了自增函数，就可以在它的基础上继续定义加法与乘法函数：</p>
<ul>
<li>PLUS = <code>λm.λn.λf.λx.m f (n f x)</code> <strong> n+m </strong>可以看做是在n的基础上调用m次自增函数</li>
<li>MULT = <code>λm.λn.m (PLUS n) 0</code> <strong> m*n </strong>可以看做是使用丘奇数0调用m次加n函数</li>
</ul>
<h3 id="四-总结"><a href="#四-总结" class="headerlink" title="四.总结"></a>四.总结</h3><p>上面演示的功能相当的naive，但从中已经可以领略到λ演算的思想，即先定义最简单的lambda term(函数)，之后通过将简单的表达式不断的组合来实现复杂的算法。下篇文章我打算继续学习如何使用λ演算实现递归函数，换句话说如何使用匿名函数来实现递归?</p>
<p>参考链接：</p>
<ul>
<li><a href="https://en.wikipedia.org/wiki/Lambda_calculus">Lambda_calculus</a></li>
<li><a href="http://goodmath.blogspot.hk/2006/05/my-favorite-calculus-lambda-part-1.html">My Favorite Calculus: Lambda (part 1)</a></li>
<li><a href="http://www.yinwang.org/blog-cn/2015/10/18/turing">图灵的光环</a></li>
</ul>
]]></content>
      
        <categories>
            
            <category> Functional Programming </category>
            
        </categories>
        
        
        <tags>
            
            <tag> Lambda Calculus </tag>
            
        </tags>
        
    </entry>
    
    <entry>
      <title><![CDATA[加密算法简介]]></title>
      <url>https://blog.staynoob.cn/post/2017/03/%25E5%258A%25A0%25E5%25AF%2586%25E7%25AE%2597%25E6%25B3%2595%25E7%25AE%2580%25E4%25BB%258B/</url>
      <content type="html"><![CDATA[<blockquote>
<p>最近听到一个来公司面试的码农说他使用Base64加密数据（Base64是一种编码格式，在编码解码计算过程中不会丢失信息量。通俗点说就相当于把“你好”加密成了“hello”），这跟明文没什么区别，如果一个hacker有本事截获你的数据，你不可能指望他不会解码Base64。这篇文章希望能分享一些关于加密的常识。</p>
</blockquote>
<h3 id="一-散列函数-Hash-function"><a href="#一-散列函数-Hash-function" class="headerlink" title="一.散列函数(Hash function)"></a>一.散列函数(Hash function)</h3><h4 id="1-常见误区"><a href="#1-常见误区" class="headerlink" title="1.常见误区"></a>1.常见误区</h4><p>关于加密算法，最常见的一个误区在于认为MD5,SHA1,SHA256就是加密算法，其实它们只是用来实现加密算法的一部分，更准确的说，它们只是散列函数，就像java中随处可见的<strong>hashCode</strong>方法一样，它们的共同点在于都要尽可能避免冲突（两个不同的输入却有相同的输出），而区别在于java中的hashCode方法用于提高散列表的性能，因此主要关注于计算速度而不是安全。<br>第二个常见误区在于，大家都觉得MD5,SHA1之所以安全，是因为它们计算过程不可逆，举个例子:<br><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">MD5(&quot;hello&quot;) = 5D41402ABC4B2A76B9719D911017C592</span><br></pre></td></tr></table></figure><br>对MD5散列函数输入字符串&quot;hello&quot;，总是可以得到<em>5D41402ABC4B2A76B9719D911017C592</em>，但<strong>现在没有以后也不可能有</strong>任何方法将<em>5D41402ABC4B2A76B9719D911017C592</em>解密为&quot;hello&quot;（目前所有所谓的解密都是基于猜测）。乍一看这很神奇，在解释它之前，先来看下维基百科上对散列函数的定义：</p>
<blockquote>
<p>散列函数是一种输入<strong>任意</strong>大小的数据，返回固定大小的值的函数</p>
</blockquote>
<p>继续以MD5为例，对它输入任意大小的数据，它的返回值都是128位(16字节)。假如这个过程可逆，那么无损压缩就不再有极限，也就意味着给你一张软盘，你就能存储全世界！换句话说，按照上面的定义，<strong>根本不存在可逆的散列函数</strong>。看到这里你可能依然很困惑，对于任意大小的数据散列不可逆也许比较好理解，但是&quot;hello&quot;明明只有5个字符，为什么也不可逆呢？专业一点解释就是，在计算过程中丢掉一部分信息量即可（对于信息的量化，可以参考我这篇文章<a href="http://blog.staynoob.cn/post/2016/05/shannon-mathematical-theory-of-communication/">香农信息论与回答老鼠喝药问题的正确姿势</a>），草根一点的解释是，你只要让你的函数对于不同的输入，可以得到相同的输出即可。比如f(x)=x^2就是一个不可逆的函数，因为已知f(x)=1，没办法知道x到底是正1还是负1。</p>
<h4 id="2-散列冲突-Hash-collision"><a href="#2-散列冲突-Hash-collision" class="headerlink" title="2.散列冲突(Hash collision)"></a>2.散列冲突(Hash collision)</h4><p>对于任意输入数据，经过散列函数计算后得到的散列值，又称为数据指纹（fingerprint）或摘要(digest)，它就像人类的指纹一样，你不可能通过一个人的指纹而了解这个人的全部信息，但是你总是可以快速的取到一个人的指纹。并且很难重复，如果两个不同的输入数据有着同样的散列值，则称为散列冲突。正常情况下假设一个结果分布完全均匀的散列函数输出128位的散列值，那么发生冲突的概率大概在$1/2^{128}$这个数量级，这几乎等同于不可能。但人们可以根据散列函数的实现过程，设计算法来查找冲突值，也就是常说的碰撞攻击(collision attack)。加密用散列函数最大的设计难点，就在于要让别人即使知道了该函数的所有实现细节，也无法找到高效的碰撞攻击算法。</p>
<h4 id="3-过时的散列函数-Deprecated-hash-function"><a href="#3-过时的散列函数-Deprecated-hash-function" class="headerlink" title="3.过时的散列函数(Deprecated hash function)"></a>3.过时的散列函数(Deprecated hash function)</h4><p>2004年山东大学的王小云教授宣布<strong>攻破MD4,MD5</strong>(<a href="http://eprint.iacr.org/2004/199.pdf">Collisions for Hash Functions</a>)，2017年（也就是最近）荷兰密码学研究小组与Google宣布<strong>攻破SHA-1</strong>(<a href="https://security.googleblog.com/2017/02/announcing-first-sha1-collision.html">Announcing the first SHA1 collision</a>，<a href="http://shattered.it/">shattered</a>)，下图解释了对这些函数查找散列冲突所需的计算量：<br><img src="/img/content/hash-collision-computation.jpg" alt="hash-collision-computation"><br>可以看到使用google最近研究的算法破解SHA-1需要<strong>110</strong>块GPU一年的计算量，虽然这个数字看起来非常大，不过依然比暴力算法快了十万倍，因此Google已将SHA-1算法标记为过时(deprecated)的（顺便提一句，linus(linux,git的开发者)最近也受到这条新闻的压力，发文宣布<a href="https://plus.google.com/+LinusTorvalds/posts/7tp2gYWQugL">更新git中用到的SHA-1算法</a>）。当然上图还有一点值得注意，前文提到的MD5算法，对它进行碰撞攻击只需要消耗一台智能手机<strong>30秒</strong>的计算量。</p>
<span id="more"></span>
<h3 id="二-如何攻击加密算法"><a href="#二-如何攻击加密算法" class="headerlink" title="二.如何攻击加密算法"></a>二.如何攻击加密算法</h3><h4 id="1-暴力算法-Brute-force-attacks"><a href="#1-暴力算法-Brute-force-attacks" class="headerlink" title="1.暴力算法(Brute force attacks)"></a>1.暴力算法(Brute force attacks)</h4><p>已知密码的散列值（数据指纹），和它的加密算法，如何破解该密码?最简单的办法就是穷举该密码所有可能的字符组合，对每个结果调用加密算法，然后与已知的散列值比对。这种办法需要消耗巨大的计算量，但它的好处是总会找到该密码的散列冲突值。举例来说，假设某地警察局获取了罪犯留在犯罪现场的指纹，不可能仅根据该指纹就解析出罪犯的所有信息。于是警察想了一个办法，让当地所有人到警察局一一采集指纹进行比对。这就是暴力算法。</p>
<h4 id="2-字典攻击-Dictionary-attacks"><a href="#2-字典攻击-Dictionary-attacks" class="headerlink" title="2.字典攻击(Dictionary attacks)"></a>2.字典攻击(Dictionary attacks)</h4><p>字典攻击指的是先使用一个文件，记录常用来作为密码的单词，这些单词可以从已经攻破的生产环境数据库提取，或者使用一些机器学习算法生成。然后使用这些词的散列值与已知密码的散列值进行比对来破解密码。继续用上面的例子，字典攻击就类似于警察局先在所有人中挑出一部分有犯罪记录的人，只采集这些人的指纹来查找罪犯。没有任何方法可以防御暴力攻击与字典攻击，换句话说如果一个加密算法是安全的，那么它只能被暴力算法和字典攻击来破解。</p>
<h4 id="3-查找表-Lookup-Tables"><a href="#3-查找表-Lookup-Tables" class="headerlink" title="3.查找表(Lookup Tables)"></a>3.查找表(Lookup Tables)</h4><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">Searching: 5f4dcc3b5aa765d61d8327deb882cf99: FOUND: password5</span><br><span class="line">Searching: 6cbe615c106f422d23669b610b564800:  not in database</span><br><span class="line">Searching: 630bf032efe4507f2c57b280995925a9: FOUND: letMEin12</span><br><span class="line">Searching: 386f43fab5d096a7a66d67c8f213e5ec: FOUND: mcd0nalds</span><br><span class="line">Searching: d5ec75d5fe70d428685510fae36492d9: FOUND: p@ssw0rd!</span><br></pre></td></tr></table></figure>
<p>查找表是一种非常有效的破解常用散列函数的方法，它的思路就是事先计算好常见密码的散列值并存在数据库中（牺牲空间复杂度来降低时间复杂度）。好的查找表实现即便预先存储了几十亿对散列值，依然能达到每秒查找上百个散列值的效率。查找表相当于警察将有犯罪记录的人的指纹保存在数据库中，之后拿到犯罪现场的指纹，直接先从数据库中搜索比对。目前很多在线提供密码破解的网站都是基于查找表实现的。</p>
<h4 id="4-反向查找表-Reverse-Lookup-Tables"><a href="#4-反向查找表-Reverse-Lookup-Tables" class="headerlink" title="4.反向查找表(Reverse Lookup Tables)"></a>4.反向查找表(Reverse Lookup Tables)</h4><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">Searching for hash(apple) in users&#x27; hash list...     : Matches [alice3, 0bob0, charles8]</span><br><span class="line">Searching for hash(blueberry) in users&#x27; hash list... : Matches [usr10101, timmy, john91]</span><br><span class="line">Searching for hash(letmein) in users&#x27; hash list...   : Matches [wilson10, dragonslayerX, joe1984]</span><br><span class="line">Searching for hash(s3cr3t) in users&#x27; hash list...    : Matches [bruce19, knuth1337, john87]</span><br><span class="line">Searching for hash(z@29hjja) in users&#x27; hash list...  : No users used this password</span><br></pre></td></tr></table></figure>
<p>顾名思义，该方法根据要破解的用户表来建立查找表，比如说我现在拿到了一张用户表，其中每个用户的密码都是散列值。我可以先遍历这张表，将其存到一个HashMap结构中，使用密码散列值作为key，使用用户名数组作为value。之后我可以随便猜一些常用密码比如123456，在HashMap中查找键值<strong>Hash(123456)</strong>就能得到所有密码是123456的用户名。</p>
<h4 id="5-彩虹表-Rainbow-Tables"><a href="#5-彩虹表-Rainbow-Tables" class="headerlink" title="5.彩虹表(Rainbow Tables)"></a>5.彩虹表(Rainbow Tables)</h4><p>假设有一个散列函数H，和一个有限的密码集合P，目标是构建一张表，使得对于任意散列值h，可以找到P中对应的元素p满足<code>H(p)=h</code>，或者可以判定该p在P集合中不存在。最简单的实现是计算P中所有元素的散列值并存储下来。这样就能得到一个查找表，这种方法的问题在于假如P集合非常大，则需要耗费巨大的存储空间。<br>另一种思路是引入一个归约函数R(reduction function)，这个归约函数输入一个散列值h，返回一个字符串（并不要求H(p)=h），通过交替应用散列函数H与归约函数R，可以形成一个由密码与散列值组成的散列链。举个栗子，如果P是所有小写字母与数字的集合，散列值为32位，那么一个散列链大概长这样：<br><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">aaaaaa -&gt; 281DAF40 -&gt; shfnyd -&gt; 920ECF10 -&gt; kiebgt</span><br></pre></td></tr></table></figure><br>我们可以选定P中的一个集合，计算多条这样的散列链，并且约定每条散列链的长度k。之后只保存散列链的起点与终点，如上例中的aaaaaa与kiebgt。现在假设我们要查找一个散列值h对应的p，可以先对h调用归约函数R，之后同样是交替调用H与R，直到中途得到一个值与我们存储的一个终点相匹配。再利用该终点与对应的起点重新生成散列链，该链中就很有可能包含散列值h，于是可以立即得到h散列前的值p。<br>举例来说，假设我们要查找散列值920ECF10对应的密码p，先对它调用归约函数得到kiebgt，之后发现该结果是已保存散列链的终点。于是利用该终点与对应的起点aaaaaa重新计算得到<strong>aaaaaa -&gt; 281DAF40 -&gt; shfnyd -&gt; 920ECF10 -&gt; kiebgt</strong>。于是得到了920ECF10散列前的值<strong>shfnyd</strong>。<br>上面演示的是散列链在理想情况下的工作方式，实际应用中还会碰到很多问题。其中有一个严重的缺陷在于如果两条链中的任何两个点碰撞（有同样的值），那它们后续的所有点都将重合，这将浪费很大的计算量。彩虹表就是用来解决该问题的，它采用一系列归约函数$R_1,R_2...R_k$来代替上面的归约函数R。最终实现如果两个散列链发生碰撞，那它们的终点一定相同。这样就可以依据终点来删除重复的散列链，从而大幅降低了碰撞的次数。<br>简而言之，暴力算法需要计算每一个可能的猜测，意味着消耗大量的时间。查找表需要保存每一个可能的猜测，意味着消耗大量的存储。彩虹表是处于它们之间的一个折中的解决方案。</p>
<h3 id="三-关于加盐-salt"><a href="#三-关于加盐-salt" class="headerlink" title="三.关于加盐(salt)"></a>三.关于加盐(salt)</h3><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">hash(&quot;hello&quot;)                    = 2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824</span><br><span class="line">hash(&quot;hello&quot; + &quot;QxLUF1bgIAdeQX&quot;) = 9e209040c863f84a31e719795b2577523954739fe5ed3b58a75cff2127075ed1</span><br><span class="line">hash(&quot;hello&quot; + &quot;bv5PehSMfV11Cd&quot;) = d1d3ec2e6f20fd420d50e2642992841d8338a314b8ea157c9e18477aaef226ab</span><br><span class="line">hash(&quot;hello&quot; + &quot;YYLmfY6IehjZMQ&quot;) = a49670c3c18b9e079b9cfaf51634f563dc8ae3070db2c4a8544305df1b60f007</span><br></pre></td></tr></table></figure>
<p>由于散列函数对同样的密码总是会得到同样的散列值，因此查找表与彩虹表才能够起效。如果每次在密码散列之前，对该密码添加一个随机字符串。就可以有效的防止这种攻击，而这个随机字符串就称为盐值(salt)。如上面的例子，在对hello进行散列之前添加一个随机的盐值，就可以保证每次加密hello都能得到不同的结果。但这样一来每次生成密码散列值后，还需要保存对应的盐值，否则之后当用户输入密码登陆时，就没办法验证该密码的正确性了。下面列一些关于盐值的常识：</p>
<ul>
<li>盐值没必要保密<br>  由于攻击者事先不知道盐值，就没办法事先计算查找表与彩虹表，又因为相同密码每次得到的散列值都不一样，因此反向查找表也没办法起效。这就是盐值存在的目的。使用任意足够长的随机字符串都可以达到该目的，而试图将该字符串保密则完全是多此一举。<br>  一般来说可以把盐值保存在用户表中，或者直接与密码的散列值保存在一起就可以了。</li>
<li>不要重用盐值<br>  一个常见的错误是只随机生成一次盐值，之后就所有密码散列都重用该盐值，或者干脆将盐值作为常量写死在程序中。这样一来攻击者可以根据该盐值来计算查找表与彩虹表。同时由于两个相同的密码还是会得到相同的散列值。反向查找表也依然有效，攻击者只要在每次猜测密码后，将该盐值加到猜测的密码上就可以了。<br>  正确的做法是每次要生成密码散列值的时候，都为该密码生成一个新的随机的盐值。</li>
<li>不要使用太短的盐值<br>  如果盐值太短，攻击者依然可以通过枚举所有可能盐值的方式来构建查找表，举例来说如果盐值只包含三个ASCII字符，那么一共只有95^3种可能的盐值。如果一个查找表包含1MB最常见的密码，那么一个包含所有盐值的查找表大概是837GB，考虑现在花300来块就能买个1TB的硬盘，这个数值真的不算大。<br>  选择多长的盐值合适？一个简单的方法是跟散列值一样长就好，比如说SHA256的输出结果为256bits(32字节)，那么使用32字节的盐值就可以了。</li>
</ul>
<h3 id="四-算得慢一点！"><a href="#四-算得慢一点！" class="headerlink" title="四.算得慢一点！"></a>四.算得慢一点！</h3><p>前面提到使用暴力算法来查找SHA-1散列函数的冲突值，需要耗费数千万块GPU一年的运算量。这可能会让你产生一些错觉，因为它是不限定输入的。换句话说假如限定输入只由6位数字与小写字母组成，那么一块GPU每秒可以破解数十亿个这样的输入。对于实际场景中一些比较复杂的密码，其计算效率也不会低于每秒一个。这也解释了为什么说SHA-256,SHA-512是安全的散列函数，但不是安全的加密算法。因为使用暴力算法破解这些安全的散列函数一样是非常高效的。<br>解决的办法是使用一种称为<a href="https://en.wikipedia.org/wiki/Key_stretching">key stretching</a>的技术，它的用途就是让散列函数变慢。以bcrypt为例，该算法接受一个安全因子(security factor)为参数，使用该参数来决定密码散列过程有多慢。通过调整该参数，可以将散列函数计算时间控制在0.3到0.5秒之间。这样的时间消耗对用户来说几乎感受不到。但是对于那些试图通过暴力算法来攻击系统的人来说，是慢得无法忍受的。举例来说，假设密码只由6位数字与小写字母组成，一台现代PC使用MD5算法枚举完所有可能性所花的时间不超过30秒。但是如果你将一次散列函数的计算时间控制在0.5秒左右，那么将密码枚举完大概要30年。</p>
<h3 id="五-Sample"><a href="#五-Sample" class="headerlink" title="五.Sample"></a>五.Sample</h3><p>bcrypt是目前比较成熟的一种加密算法，下面通过分析一个它的散列值，来看看它涉及到上文提到的哪些要点。<br><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$2a$10$N9qo8uLOickgx2ZMRZoMyeIjZAgcfl7p92ldGxad68LJZdL17lhWy</span><br></pre></td></tr></table></figure><br>该散列值格式称为Modular Crypt Format，其中包括如下内容：</p>
<ul>
<li>id:2a<br>  2a是一个标识符，用于指示用到的散列函数，如1表示MD5，5表示SHA-256</li>
<li>param:10<br>  这里的10，就是上面提到的安全因子。该值越大，表示用于生成该散列值的函数运行速度越慢。</li>
<li>salt:N9qo8uLOickgx2ZMRZoMye<br>  128位盐值，使用Base64编码为22个字符</li>
<li>hash:IjZAgcfl7p92ldGxad68LJZdL17lhWy<br>  实际密码与盐值共同散列后的结果，占184位，使用Base64编码为31个字符</li>
</ul>
<p>最后，这些常识只能帮你鉴别哪些加密不安全，或者说不够安全。了解了这些内容并不意味着你可以在生产环境中使用你自己写的加密算法，本着对用户负责的态度，应该始终选择经过考验的成熟算法。</p>
<p>参考链接：</p>
<ul>
<li><a href="https://crackstation.net/hashing-security.htm">hashing-security</a></li>
<li><a href="https://en.wikipedia.org/wiki/Bcrypt">Bcrypt</a></li>
<li><a href="https://en.wikipedia.org/wiki/Rainbow_table">rainbow table</a></li>
</ul>
]]></content>
      
        <categories>
            
            <category> Security </category>
            
        </categories>
        
        
    </entry>
    
    <entry>
      <title><![CDATA[Bash小技巧]]></title>
      <url>https://blog.staynoob.cn/post/2017/02/bash%25E5%25B0%258F%25E6%258A%2580%25E5%25B7%25A7/</url>
      <content type="html"><![CDATA[<blockquote>
<p>bash的全称是<em>Bourne Again Shell</em>，是大多数<em>linux</em>发行版以及<em>OSX</em>系统的默认shell，<em>win10</em>可以开启开发者模式来使用bash，因此bash是目前所有主流操作系统中都可以使用的命令行环境。</p>
</blockquote>
<h3 id="一-写在前面"><a href="#一-写在前面" class="headerlink" title="一.写在前面"></a>一.写在前面</h3><p>这篇文章只是介绍一些非常好用，但我总是记不住的bash小技巧。按照惯例，先来做个小测验，如果你对<code>!!</code>,<code>!-1</code>,<code>!$</code>,<code>!*</code>分表代表什么了然于心，我想你应该不需要读这篇文章。</p>
<span id="more"></span>
<h3 id="二-快捷键"><a href="#二-快捷键" class="headerlink" title="二.快捷键"></a>二.快捷键</h3><p>bash快捷键默认使用的是<a href="http://cnswww.cns.cwru.edu/php/chet/readline/rltop.html">readline</a>库的<a href="https://zh.wikipedia.org/wiki/Emacs">Emacs</a>模式，因此如果你是Emacs党，可以直接跳过这一节。</p>
<ul>
<li><p>历史命令<br>  一个常见的应用场景是你需要重试之前输入过的某条命令，常见的做法是不停的按上下箭头来找出那条命令，我个人则比较喜欢ctrl+p(arrow up)与ctrl+n(arrow down)，也可以通过<code>history</code>配合<code>grep</code>来查找命令。而这里推荐的方法是使用Ctrl+r，然后输入你想要的命令中含有的单词，之后会出现含有这个单词的命令，如果它不是你想要的命令，就继续按Ctrl+r，直到找到你要的命令为止。中途你还可以继续输入查找字符，总之谁用谁知道。</p>
</li>
<li><p>移动光标<br>  常见的应用场景是洋洋洒洒敲了一大串命令，结果发现命令中有几个字母输错了，又或者是忘了加<code>sudo</code>，这时候常见的做法是按<strong>arrow left</strong>键移动光标到指定的位置，当然也不妨尝试下面的快捷键。</p>
<ul>
<li><strong>ctrl+b</strong> 相当于arrow left</li>
<li><strong>ctrl+f</strong> 相当于arrow right</li>
<li><strong>alt+b</strong> 向前移动一个单词</li>
<li><strong>alt+f</strong> 向后移动一个单词</li>
<li><strong>ctrl+a</strong> 移至行首，相当于home</li>
<li><strong>ctrl+e</strong> 移至行尾，相当于end</li>
<li><strong>ctrl+xx</strong> 在命令行首和光标当前位置之间移动</li>
</ul>
</li>
</ul>
<ul>
<li>编辑<br>  像我这种手残党使用backspace删除单词时，经常不小心多按或者少按，这时候就体现出按单词删除的优越性了，但是我不仅手残，脑袋也不怎么灵光，ctrl+w与alt+d的效果经常弄混有没有?没关系，可以使用ctrl+/来撤销编辑了。下面列出我觉得比较有价值的编辑快捷键。<ul>
<li><strong>ctrl+w</strong> 向前删除一个单词</li>
<li><strong>alt+d</strong> 向后删除一个单词</li>
<li><strong>ctrl+k</strong> 删除当前光标到行尾的字符</li>
<li><strong>ctrl+u</strong> 删除当前光标到行首的字符</li>
<li><strong>alt+u</strong> 将字母转为大写(光标处至词尾)</li>
<li><strong>alt+l</strong> 将字母转为小写(光标处至词尾)</li>
<li><strong>alt+.</strong> 获取上条命令最后一个参数</li>
<li><strong>ctrl+/</strong> 撤销</li>
</ul>
</li>
</ul>
<h3 id="三-历史扩展-History-Expansion"><a href="#三-历史扩展-History-Expansion" class="headerlink" title="三.历史扩展(History Expansion)"></a>三.历史扩展(History Expansion)</h3><p>Bash 的历史扩展又被称为 Bang(!) 命令，是 bash 将历史命令转换到可执行命令的过程。历史扩展中操作历史命令一般有两个部分，即先从历史命令中找出相应的命令，被选中的命令称为Event，bash会将选中的Event以空格分割成Words，之后再使用单词指示器选择部分或全部的Word到当前行中。</p>
<ul>
<li>事件指示器(Event Designators)<br>  这次我们又行云流水的敲了一大串命令，自信回车一气呵成，结果系统提示“权限不够”，是不是很尴尬？通过上面的介绍，想必你应该马上能想到使用<strong>ctrl+p</strong>翻出上一条命令，再使用<strong>ctrl+a</strong>将光标移至行首，添加<strong>sudo</strong>执行。不过这里还有更加优雅的方式，直接执行<code>sudo !!</code>，这里<strong> !! </strong>中的第一个感叹号代表开始历史替换，而第二份感叹号则是事件指示器，下面有更详细的用法。<ul>
<li><strong> n </strong> 选择历史命令列表中第n条命令</li>
<li><strong> -n </strong> 选择倒数第n条命令</li>
<li><strong> ! </strong> 选择上一条命令, 相当于<strong>-1</strong></li>
<li><strong> string </strong> 选择最近的以string开头的命令</li>
<li><strong> ?string[?] </strong> 选择最近的包含string的命令, 如果该指示器后面是换行符, 则可以不用输入结尾的<strong>[?]</strong></li>
<li><strong> # </strong> 引用目前输入的所有命令</li>
</ul>
</li>
</ul>
<ul>
<li>单词指示器(Word Designators)<br>  有时候我们只想要获取上条命令的参数，而不包含命令本身。比如说我先执行了  <figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">bower install react react-dom ...(此处省略100多个库你懂得) --save</span><br></pre></td></tr></table></figure>
  之后发现自己又sb了，应该改用npm来执行同样的命令，这时可以输入  <figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">npm !*</span><br></pre></td></tr></table></figure>
  其效果等同于  <figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">npm install react react-dom ... --save</span><br></pre></td></tr></table></figure>
  这里<strong> !* </strong>中的<strong> * </strong>就是单词指示器，也许你发现了这里并没有指定事件，如果没有事件指示器，则单词指示器的操作条目默认是上一条命令，因此<strong> !* </strong>其实相当于<strong> !!* </strong>。一般情况下事件指示器与单词指示器需要以<strong> : </strong>分隔，但如果单词指示器是以<strong> ^,$,*,-,% </strong>开头时，可以省略<strong> : </strong>，因此<strong> !* </strong>的完整形式是<strong> !!:* </strong>，下面是其它单词指示器。<ul>
<li><strong> n </strong> 选择第n个word，如果是0则选择的是命令</li>
<li><strong> ^ </strong> 选择第一个参数，相当于1</li>
<li><strong> $ </strong> 选择最后一个word</li>
<li><strong> n1-n2 </strong> 选择第n1到第n2个word</li>
<li><strong> * </strong> 选择命令的所有参数, 相当于1-$</li>
<li><strong> n* </strong> 相当于n-$</li>
</ul>
</li>
</ul>
<ul>
<li>修饰符(Modifiers)<br>  在单词指示器后面其实还可以添加修饰符，同样以<strong> : </strong>分隔，比如<code>!!:*:p</code>会打印上条命令的所有参数，但不会执行。这个特性我不是很常用，为了合理(lang)利用(de)脑容量(ji)，这里就不罗列了，有兴趣的读者可以在参考链接中查看更详细的内容。</li>
</ul>
<p>参考链接：</p>
<ul>
<li><a href="http://ahei.info/bash.htm">高效操作Bash</a></li>
<li><a href="https://linux.cn/article-5658-1.html">感叹号：bash 的历史扩展功能</a></li>
</ul>
]]></content>
      
        <categories>
            
            <category> 永远记不住的编程姿势 </category>
            
        </categories>
        
        
        <tags>
            
            <tag> Linux </tag>
            
            <tag> 编程技巧 </tag>
            
        </tags>
        
    </entry>
    
    <entry>
      <title><![CDATA[正则表达式：零宽断言(lookaround)]]></title>
      <url>https://blog.staynoob.cn/post/2017/01/%25E6%25AD%25A3%25E5%2588%2599%25E8%25A1%25A8%25E8%25BE%25BE%25E5%25BC%258F-%25E9%259B%25B6%25E5%25AE%25BD%25E6%2596%25AD%25E8%25A8%2580-lookaround/</url>
      <content type="html"><![CDATA[<blockquote>
<p>零宽断言(lookaround/zero-width assersion)是正则表达式中非常常用的特性，但是因为其一些带误导性的中文翻译和易于混淆的写法，让我一度每次使用都要先查文档。</p>
</blockquote>
<h3 id="一-写在前面"><a href="#一-写在前面" class="headerlink" title="一.写在前面"></a>一.写在前面</h3><p>这篇文章讲的东西非常基础，之所以要写一遍完全是因为我智商经常离线，老把它们写混。如果你对下面五种模式及其用途了然于心，请不要浪费时间读这篇文章。</p>
<ul>
<li><code>(?:pattern)</code>非捕获匹配(matches pattern but does not capture the match)</li>
<li><code>(?=pattern)</code>零宽正向先行断言(lookahead)</li>
<li><code>(?!pattern)</code>零宽负向先行断言(negative lookahead)</li>
<li><code>(?&lt;=pattern)</code>零宽正回顾后发断言(lookbehind)</li>
<li><code>(?&lt;!pattern)</code>零宽负回顾后发断言(negative lookbehind)</li>
</ul>
<p>之所以我要在每种模式后打出它的英文表达方式，不是为了装B，而是想让大家感受一下这个翻译(msdn,javascript权威指南等多本书籍或文档采用这样，或者类似这样的翻译)。我反正是第一次体会到英文看的明明白白，中文看得一头雾水的感觉。</p>
<span id="more"></span>
<h3 id="二-功能介绍"><a href="#二-功能介绍" class="headerlink" title="二.功能介绍"></a>二.功能介绍</h3><p>相比理解这些模式的中文名，理解它们的用法要简单得多。（更多详情可以参见文章最后的测试用例）</p>
<ul>
<li><code>(?:a)(b)\\1</code>匹配abb，如果是<code>(a)(b)\\1</code>则匹配aba，<code>(?:pattern)</code>表示匹配但不捕获，可以将其理解成不创建分组的<code>(pattern)</code>。</li>
<li><code>a(?=b)</code>断言a后面有b，匹配ab但不匹配aa，最终捕获a。（个人认为应该翻译成<em>正向肯定查找</em>）</li>
<li><code>a(?!b)</code>断言a后面没有b，匹配aa但不匹配ab，最终捕获a。（个人认为应该翻译成<em>正向否定查找</em>）</li>
<li><code>(?&lt;=a)b</code>断言b前面有a，匹配ab但不匹配bb，最终捕获b。（个人认为应该翻译成<em>反向肯定查找</em>）</li>
<li><code>(?&lt;!a)b</code>断言b前面没有a，匹配bb但不匹配ab，最终捕获b。（个人认为应该翻译成<em>反向否定查找</em>）</li>
</ul>
<h3 id="三-Polyfill"><a href="#三-Polyfill" class="headerlink" title="三.Polyfill"></a>三.Polyfill</h3><p>lookbehind目前还处在es7的<a href="https://github.com/goyakin/es-regexp-lookbehind">提案</a>阶段，因此无法在js中直接使用（也正因为如此，VSC的查找与替换不支持lookbehind），新版的V8引擎已经实现了该功能。不过需要在浏览器设置中打开“#enable-javascript-harmony”开关（在chrome地址栏输入<em>about:flags</em>设置）<br>在js实现该功能之前，可以使用捕获与引用来达成类似的效果，举例来说如果你需要将所有前面带有static修饰符的int，替换成long，在支持lookbehind的编辑器中可以直接查找<code>(?&lt;=static\s)int</code>，并将其替换成<code>long</code>。而在不支持lookbehind的环境下（vscode别瞅了，说的就是你！），可以查找<code>(static\s)int</code>，将其替换成<code>$1long</code>。</p>
<h3 id="四-Code-Demo"><a href="#四-Code-Demo" class="headerlink" title="四.Code Demo"></a>四.Code Demo</h3><p>Polyfill Demo(js)<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">//测试依赖mocha,shouldjs</span></span><br><span class="line"><span class="title function_">describe</span>(<span class="string">&#x27;polyfill&#x27;</span>, <span class="keyword">function</span> (<span class="params"></span>) &#123;</span><br><span class="line">    <span class="title function_">it</span>(<span class="string">&#x27;使用捕获与引用来替代lookbehind&#x27;</span>, <span class="keyword">function</span> (<span class="params"></span>) &#123;</span><br><span class="line">        <span class="string">&#x27;static int&#x27;</span>.<span class="title function_">replace</span>(<span class="regexp">/(static\s)int/</span>, <span class="string">&#x27;$1long&#x27;</span>)</span><br><span class="line">            .<span class="property">should</span>.<span class="title function_">equal</span>(<span class="string">&#x27;static long&#x27;</span>);</span><br><span class="line">    &#125;);</span><br><span class="line">&#125;);</span><br></pre></td></tr></table></figure></p>
<p>Feature Demo(java)<br><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">//测试依赖junit4,assertj</span></span><br><span class="line"><span class="meta">@Test</span></span><br><span class="line"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title function_">matchButNotCapture</span><span class="params">()</span> <span class="keyword">throws</span> Exception &#123;</span><br><span class="line">    <span class="type">String</span> <span class="variable">exp</span> <span class="operator">=</span> <span class="string">&quot;(?:a)(b)\\1&quot;</span>;</span><br><span class="line">    <span class="comment">//由于(?:)不创建引用，因此\1引用了(b)捕获的字符串</span></span><br><span class="line">    Assertions.assertThat(Pattern.matches(exp, <span class="string">&quot;aba&quot;</span>)).isFalse();</span><br><span class="line">    Assertions.assertThat(Pattern.matches(exp, <span class="string">&quot;abb&quot;</span>)).isTrue();</span><br><span class="line">    <span class="comment">//只有一个分组</span></span><br><span class="line">    Assertions.assertThat(Pattern.compile(exp).matcher(<span class="string">&quot;abb&quot;</span>).groupCount()).isEqualTo(<span class="number">1</span>);</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="meta">@Test</span></span><br><span class="line"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title function_">lookahead</span><span class="params">()</span> <span class="keyword">throws</span> Exception &#123;</span><br><span class="line">    <span class="comment">//匹配后面有b的a</span></span><br><span class="line">    <span class="type">String</span> <span class="variable">exp</span> <span class="operator">=</span> <span class="string">&quot;a(?=b)&quot;</span>;</span><br><span class="line">    Assertions.assertThat(Pattern.compile(exp).matcher(<span class="string">&quot;aa&quot;</span>).find()).isFalse();</span><br><span class="line">    <span class="comment">//zero-width</span></span><br><span class="line">    <span class="type">Matcher</span> <span class="variable">matcher</span> <span class="operator">=</span> Pattern.compile(exp).matcher(<span class="string">&quot;ab&quot;</span>);</span><br><span class="line">    Assertions.assertThat(matcher.find()).isTrue();</span><br><span class="line">    Assertions.assertThat(matcher.group()).isEqualTo(<span class="string">&quot;a&quot;</span>);</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="meta">@Test</span></span><br><span class="line"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title function_">negativeLookahead</span><span class="params">()</span> <span class="keyword">throws</span> Exception &#123;</span><br><span class="line">    <span class="comment">//匹配后面没有b的a</span></span><br><span class="line">    <span class="type">String</span> <span class="variable">exp</span> <span class="operator">=</span> <span class="string">&quot;a(?!b)&quot;</span>;</span><br><span class="line">    Assertions.assertThat(Pattern.compile(exp).matcher(<span class="string">&quot;ab&quot;</span>).find()).isFalse();</span><br><span class="line">    <span class="comment">//zero-width</span></span><br><span class="line">    <span class="type">Matcher</span> <span class="variable">matcher</span> <span class="operator">=</span> Pattern.compile(exp).matcher(<span class="string">&quot;aa&quot;</span>);</span><br><span class="line">    Assertions.assertThat(matcher.find()).isTrue();</span><br><span class="line">    Assertions.assertThat(matcher.group()).isEqualTo(<span class="string">&quot;a&quot;</span>);</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="meta">@Test</span></span><br><span class="line"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title function_">lookbehind</span><span class="params">()</span> <span class="keyword">throws</span> Exception &#123;</span><br><span class="line">    <span class="comment">//匹配前面有a的b</span></span><br><span class="line">    <span class="type">String</span> <span class="variable">exp</span> <span class="operator">=</span> <span class="string">&quot;(?&lt;=a)b&quot;</span>;</span><br><span class="line">    Assertions.assertThat(Pattern.compile(exp).matcher(<span class="string">&quot;bb&quot;</span>).find()).isFalse();</span><br><span class="line">    <span class="comment">//zero-width</span></span><br><span class="line">    <span class="type">Matcher</span> <span class="variable">matcher</span> <span class="operator">=</span> Pattern.compile(exp).matcher(<span class="string">&quot;ab&quot;</span>);</span><br><span class="line">    Assertions.assertThat(matcher.find()).isTrue();</span><br><span class="line">    Assertions.assertThat(matcher.group()).isEqualTo(<span class="string">&quot;b&quot;</span>);</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="meta">@Test</span></span><br><span class="line"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title function_">negativeLookbehind</span><span class="params">()</span> <span class="keyword">throws</span> Exception &#123;</span><br><span class="line">    <span class="comment">//匹配前面没有a的b</span></span><br><span class="line">    <span class="type">String</span> <span class="variable">exp</span> <span class="operator">=</span> <span class="string">&quot;(?&lt;!a)b&quot;</span>;</span><br><span class="line">    Assertions.assertThat(Pattern.compile(exp).matcher(<span class="string">&quot;ab&quot;</span>).find()).isFalse();</span><br><span class="line">    <span class="comment">//zero-width</span></span><br><span class="line">    <span class="type">Matcher</span> <span class="variable">matcher</span> <span class="operator">=</span> Pattern.compile(exp).matcher(<span class="string">&quot;bb&quot;</span>);</span><br><span class="line">    Assertions.assertThat(matcher.find()).isTrue();</span><br><span class="line">    Assertions.assertThat(matcher.group()).isEqualTo(<span class="string">&quot;b&quot;</span>);</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure></p>
]]></content>
      
        <categories>
            
            <category> 永远记不住的编程姿势 </category>
            
        </categories>
        
        
        <tags>
            
            <tag> 编程技巧 </tag>
            
        </tags>
        
    </entry>
    
    <entry>
      <title><![CDATA[Jackson序列化双向关联映射的四种方法]]></title>
      <url>https://blog.staynoob.cn/post/2017/01/Jackson%25E5%25BA%258F%25E5%2588%2597%25E5%258C%2596%25E5%258F%258C%25E5%2590%2591%25E5%2585%25B3%25E8%2581%2594%25E6%2598%25A0%25E5%25B0%2584%25E7%259A%2584%25E5%259B%259B%25E7%25A7%258D%25E6%2596%25B9%25E6%25B3%2595/</url>
      <content type="html"><![CDATA[<blockquote>
<p>如果一个对象与另一个对象之间存在互相引用(bi-directional relationship)，序列化该对象就会出现stackoverflow异常，这篇文章主要介绍使用jackson序列化工具处理该问题的几种解决方案。</p>
</blockquote>
<h3 id="一-Infinite-Recursion"><a href="#一-Infinite-Recursion" class="headerlink" title="一.Infinite Recursion"></a>一.Infinite Recursion</h3><p>首先不要被文章标题所误导，该问题并不是因为使用了jpa/hibernate的双向关联关系才会出现，简书上<a href="http://www.jianshu.com/p/d7f0b383a228">这篇文章</a>说该异常根本原因在于hibernate的lazyload，我认为是一本正经的胡说八道，先来看一个简单的例子:<br><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">public</span> <span class="keyword">class</span> <span class="title class_">Customer</span> &#123;</span><br><span class="line">    <span class="keyword">private</span> Long id;</span><br><span class="line">    <span class="keyword">private</span> List&lt;Card&gt; cardList;</span><br><span class="line">    <span class="comment">//constructor getter setter...</span></span><br><span class="line">&#125;</span><br><span class="line"><span class="keyword">public</span> <span class="keyword">class</span> <span class="title class_">Card</span> &#123;</span><br><span class="line">    <span class="keyword">private</span> Long id;</span><br><span class="line">    <span class="keyword">private</span> Customer customer;</span><br><span class="line">    <span class="comment">//constructor getter setter...</span></span><br><span class="line">&#125;</span><br><span class="line"><span class="comment">//测试环境 junit, assertj</span></span><br><span class="line"><span class="keyword">public</span> <span class="keyword">class</span> <span class="title class_">CustomerTest</span> &#123;</span><br><span class="line">    <span class="meta">@Test</span></span><br><span class="line">    <span class="keyword">public</span> <span class="keyword">void</span> <span class="title function_">serialize</span><span class="params">()</span> <span class="keyword">throws</span> Exception &#123;</span><br><span class="line">        <span class="type">Customer</span> <span class="variable">customer</span> <span class="operator">=</span> <span class="keyword">new</span> <span class="title class_">Customer</span>();</span><br><span class="line">        <span class="type">Card</span> <span class="variable">card</span> <span class="operator">=</span> <span class="keyword">new</span> <span class="title class_">Card</span>();</span><br><span class="line">        <span class="comment">//相互引用</span></span><br><span class="line">        card.setCustomer(customer);</span><br><span class="line">        customer.addCard(card);</span><br><span class="line">        <span class="type">ObjectMapper</span> <span class="variable">objectMapper</span> <span class="operator">=</span> <span class="keyword">new</span> <span class="title class_">ObjectMapper</span>();</span><br><span class="line">        Assertions.assertThatThrownBy(() -&gt; objectMapper.writeValueAsString(customer))</span><br><span class="line">                .hasMessageStartingWith(<span class="string">&quot;Infinite recursion (StackOverflowError)&quot;</span>);</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br>上例中消费者与信用卡之间存在双向的一对多关系，序列化其中任意一个对象都会导致StackOverflowError，换句话说只要两个对象之间存在互相引用，就无法直接序列化，即便是在js中也不例外，看代码:<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">//测试环境 mocha,shouldjs</span></span><br><span class="line"><span class="keyword">var</span> should = <span class="built_in">require</span>(<span class="string">&#x27;should&#x27;</span>);</span><br><span class="line"><span class="title function_">describe</span>(<span class="string">&#x27;json&#x27;</span>, <span class="keyword">function</span> (<span class="params"></span>) &#123;</span><br><span class="line">    <span class="title function_">it</span>(<span class="string">&#x27;如果两个对象互相引用,则无法直接序列化&#x27;</span>, <span class="keyword">function</span> (<span class="params"></span>) &#123;</span><br><span class="line">        <span class="keyword">var</span> foo = &#123;&#125;;</span><br><span class="line">        <span class="keyword">var</span> bar = &#123;&#125;;</span><br><span class="line">        foo.<span class="property">bar</span> = bar;</span><br><span class="line">        bar.<span class="property">foo</span> = foo;</span><br><span class="line">        <span class="title function_">should</span>(<span class="keyword">function</span>(<span class="params"></span>)&#123; <span class="title class_">JSON</span>.<span class="title function_">stringify</span>(foo);&#125;).<span class="keyword">throw</span>(<span class="string">&#x27;Converting circular structure to JSON&#x27;</span>);</span><br><span class="line">    &#125;);</span><br><span class="line">&#125;);</span><br></pre></td></tr></table></figure><br>综上所述，这锅hibernate不接。问题介绍完毕，下面来看看当前热门的序列化工具jackson提供了哪些方法来处理该问题，以下讨论的方法截至jackson2.8.4版本。</p>
<span id="more"></span>
<h3 id="二-JsonIgnore"><a href="#二-JsonIgnore" class="headerlink" title="二.@JsonIgnore"></a>二.@JsonIgnore</h3><p>应用JsonIgnore注解后，代码看起来像这样<br><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">public</span> <span class="keyword">class</span> <span class="title class_">Customer</span> &#123;</span><br><span class="line">    <span class="keyword">private</span> Long id;</span><br><span class="line">    <span class="keyword">private</span> List&lt;Card&gt; cardList;</span><br><span class="line">    <span class="comment">//constructor getter setter...</span></span><br><span class="line">&#125;</span><br><span class="line"><span class="keyword">public</span> <span class="keyword">class</span> <span class="title class_">Card</span> &#123;</span><br><span class="line">    <span class="keyword">private</span> Long id;</span><br><span class="line">    <span class="meta">@JsonIgnore</span></span><br><span class="line">    <span class="keyword">private</span> Customer customer;</span><br><span class="line">    <span class="comment">//constructor getter setter...</span></span><br><span class="line">&#125;</span><br><span class="line"><span class="comment">//serialize customer, result should like</span></span><br><span class="line"><span class="comment">// &#123;</span></span><br><span class="line"><span class="comment">//     &quot;id&quot;: 1,</span></span><br><span class="line"><span class="comment">//     &quot;cardList&quot;: [</span></span><br><span class="line"><span class="comment">//         &#123;</span></span><br><span class="line"><span class="comment">//             &quot;id&quot;: 1</span></span><br><span class="line"><span class="comment">//         &#125;</span></span><br><span class="line"><span class="comment">//     ]</span></span><br><span class="line"><span class="comment">// &#125;</span></span><br></pre></td></tr></table></figure><br>该注解忽略card对象的customer属性，从而避免了无限递归的问题。需要说明的是JsonIgnore注解的设计初衷并不是用于解决该问题，但由于该方法简单粗暴易于理解，因此相当一部分程序员还是选择了该方法。下面说说我不推荐使用的理由:</p>
<ol>
<li>使用了该注解后，如果你希望单独序列化一个card对象，那么json的接收端无法得知关于customer的任何信息</li>
<li>当服务端接受到客户端发送过来的json字符串，并希望反序列化为card对象时，其customer属性永远为null</li>
</ol>
<p>换言之，当你感觉某个属性需要使用JsonIgnore注解的时候，你应该先考虑是否可以直接删除该属性。拿上面的例子来说，也许你需要的只是在Card类中删除customer属性。JsonIgnore属性正确的应用场景应该是实体中类似于createAt,updateAt这样的字段，这些字段仅由服务端维护，且不需要发送至客户端，(注：password字段也不应该使用该注解，应该使用<code>@JsonProperty(access = JsonProperty.Access.WRITE_ONLY)</code>，因为你可能希望接受客户端发送的password)</p>
<h3 id="三-JsonManagedReference与-JsonBackReference"><a href="#三-JsonManagedReference与-JsonBackReference" class="headerlink" title="三.@JsonManagedReference与@JsonBackReference"></a>三.@JsonManagedReference与@JsonBackReference</h3><p>使用该方法后，代码看起来像这样<br><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">public</span> <span class="keyword">class</span> <span class="title class_">Customer</span> &#123;</span><br><span class="line">    <span class="keyword">private</span> Long id;</span><br><span class="line">    <span class="meta">@JsonManagedReference(&quot;customer-card&quot;)</span> <span class="comment">//当一个类中存在多个@JsonManagedReference注解时，其value值才是必须的</span></span><br><span class="line">    <span class="keyword">private</span> List&lt;Card&gt; cardList;</span><br><span class="line">    <span class="comment">//constructor getter setter...</span></span><br><span class="line">&#125;</span><br><span class="line"><span class="keyword">public</span> <span class="keyword">class</span> <span class="title class_">Card</span> &#123;</span><br><span class="line">    <span class="keyword">private</span> Long id;</span><br><span class="line">    <span class="meta">@JsonBackReference(&quot;customer-card&quot;)</span> <span class="comment">//同上</span></span><br><span class="line">    <span class="keyword">private</span> Customer customer;</span><br><span class="line">    <span class="comment">//constructor getter setter...</span></span><br><span class="line">&#125;</span><br><span class="line"><span class="comment">//serialize customer, result should like</span></span><br><span class="line"><span class="comment">// &#123;</span></span><br><span class="line"><span class="comment">//     &quot;id&quot;: 1,</span></span><br><span class="line"><span class="comment">//     &quot;cardList&quot;: [</span></span><br><span class="line"><span class="comment">//         &#123;</span></span><br><span class="line"><span class="comment">//             &quot;id&quot;: 1</span></span><br><span class="line"><span class="comment">//         &#125;</span></span><br><span class="line"><span class="comment">//     ]</span></span><br><span class="line"><span class="comment">// &#125;</span></span><br></pre></td></tr></table></figure><br>@JsonManagedReference与@JsonBackReference生来就是为了解决Infinite Recursion问题。它与@JsonIgnore最大的不同在于反序列化时，标注JsonIgnore注解的属性始终为null，而使用这对注解的反序列化结果为两个互相引用的对象。然而这对注解同样不推荐使用，以下是原因:</p>
<ol>
<li>相信你也看到了，它的序列化结果与使用@JsonIgnore一模一样，当你需要单独序列化card对象时，Json接收端无法知道该card属于哪个customer。</li>
<li>JsonBackReference只能使用在一对多关系中&quot;一&quot;的一端，这意味着当你有Card与CardCategory两个类时，关系的维护端在CardCategory类，序列化信用卡类别可以得到该类别的所有信用卡，而序列化信用卡却无法得到该信用卡的类别，这显然不是你想要的。</li>
</ol>
<p>综上所述，然并卵。</p>
<h3 id="四-JsonIdentityInfo"><a href="#四-JsonIdentityInfo" class="headerlink" title="四.@JsonIdentityInfo"></a>四.@JsonIdentityInfo</h3><p>使用该方法后，代码看起来像这样<br><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">@JsonIdentityInfo(generator = ObjectIdGenerators.PropertyGenerator.class, property = &quot;id&quot;, scope = Customer.class)</span></span><br><span class="line"><span class="keyword">public</span> <span class="keyword">class</span> <span class="title class_">Customer</span> &#123;</span><br><span class="line">    <span class="keyword">private</span> Long id;</span><br><span class="line">    <span class="keyword">private</span> List&lt;Card&gt; cardList;</span><br><span class="line">    <span class="comment">//constructor getter setter...</span></span><br><span class="line">&#125;</span><br><span class="line"><span class="meta">@JsonIdentityInfo(generator = ObjectIdGenerators.PropertyGenerator.class, property = &quot;id&quot;, scope = Card.class)</span></span><br><span class="line"><span class="keyword">public</span> <span class="keyword">class</span> <span class="title class_">Card</span> &#123;</span><br><span class="line">    <span class="keyword">private</span> Long id;</span><br><span class="line">    <span class="keyword">private</span> Customer customer;</span><br><span class="line">    <span class="comment">//constructor getter setter...</span></span><br><span class="line">&#125;</span><br><span class="line"><span class="comment">//为了更好的演示该注解，我在customer对象中添加了两张card</span></span><br><span class="line"><span class="comment">//serialize customer, result should like</span></span><br><span class="line"><span class="comment">// &#123;</span></span><br><span class="line"><span class="comment">//     &quot;id&quot;: 1,</span></span><br><span class="line"><span class="comment">//     &quot;cardList&quot;: [</span></span><br><span class="line"><span class="comment">//         &#123;</span></span><br><span class="line"><span class="comment">//             &quot;id&quot;: 1,</span></span><br><span class="line"><span class="comment">//             &quot;customer&quot;: 1</span></span><br><span class="line"><span class="comment">//         &#125;,</span></span><br><span class="line"><span class="comment">//         &#123;</span></span><br><span class="line"><span class="comment">//             &quot;id&quot;: 2,</span></span><br><span class="line"><span class="comment">//             &quot;customer&quot;: 1</span></span><br><span class="line"><span class="comment">//         &#125;</span></span><br><span class="line"><span class="comment">//     ]</span></span><br><span class="line"><span class="comment">// &#125;</span></span><br></pre></td></tr></table></figure><br>效果简直完美，当该注解检测到本次序列化中有重复对象时，使用指定的生成器生成一个替代对象(上例中的id)，序列化card对象不会丢失customer信息，反序列化时能够完整的将对象恢复到序列化前的状态，顺便完美的规避的数据冗余，看上去非常美好。（注：要开始装B了！）但是通信原理告诉我们，数据格式的信息冗余度越低，则对其编解码需求的运算量越大，同时在信息传输过程中，越容易受到噪声干扰而失真(参见<a href="http://blog.staynoob.cn/post/algorithm/shannon-mathematical-theory-of-communication">香农信息论与回答老鼠喝药问题的正确姿势</a>)。（注：装B结束！）下面是应用了该注解后，序列化customer.cardList的结果:<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// 理想</span></span><br><span class="line">[</span><br><span class="line">    &#123;</span><br><span class="line">        <span class="string">&quot;id&quot;</span>: <span class="number">1</span>,</span><br><span class="line">        <span class="string">&quot;customer&quot;</span>: &#123;</span><br><span class="line">            <span class="string">&quot;id&quot;</span>: <span class="number">1</span>,</span><br><span class="line">            <span class="string">&quot;cardList&quot;</span>: [<span class="number">1</span>, <span class="number">2</span>]</span><br><span class="line">        &#125;</span><br><span class="line">    &#125;,</span><br><span class="line">    &#123;</span><br><span class="line">        <span class="string">&quot;id&quot;</span>: <span class="number">2</span>,</span><br><span class="line">        <span class="string">&quot;customer&quot;</span>: &#123;</span><br><span class="line">            <span class="string">&quot;id&quot;</span>: <span class="number">1</span>,</span><br><span class="line">            <span class="string">&quot;cardList&quot;</span>: [<span class="number">1</span>, <span class="number">2</span>]</span><br><span class="line">        &#125;</span><br><span class="line">    &#125;</span><br><span class="line">]</span><br><span class="line"><span class="comment">// 现实(WTF)</span></span><br><span class="line">[</span><br><span class="line">    &#123;</span><br><span class="line">        <span class="string">&quot;id&quot;</span>: <span class="number">1</span>,</span><br><span class="line">        <span class="string">&quot;customer&quot;</span>: &#123;</span><br><span class="line">            <span class="string">&quot;id&quot;</span>: <span class="number">1</span>,</span><br><span class="line">            <span class="string">&quot;cardList&quot;</span>: [</span><br><span class="line">                <span class="number">1</span>,</span><br><span class="line">                &#123;</span><br><span class="line">                    <span class="string">&quot;id&quot;</span>: <span class="number">2</span>,</span><br><span class="line">                    <span class="string">&quot;customer&quot;</span>: <span class="number">1</span></span><br><span class="line">                &#125;</span><br><span class="line">            ]</span><br><span class="line">        &#125;</span><br><span class="line">    &#125;,</span><br><span class="line">    <span class="number">2</span></span><br><span class="line">]</span><br></pre></td></tr></table></figure><br>从上面惨不忍睹的现实中可以看出，jackson序列化对象时使用深度优先遍历(Depth-first-traversal)，在card1.customer.cardList中写入了card2的信息，后面出现的card2则被替换成其id。也许你会说丑是丑了点，但信息量并没有减少，在json的接收端经过一些特殊处理依然可以将其反序列化至原来的状态。这也正如我前面所说的，该方法增加了编码与解码所需的运算量，这意味着你可能需要为该注解引入额外的libarary来进行序列化与反序列化（例如<a href="https://github.com/jsog/jsog">JSOG</a>），考虑到json的接收端可能是各种平台(android,ios,web)，因此该方案的实用性依然非常有限。</p>
<h3 id="五-JsonView"><a href="#五-JsonView" class="headerlink" title="五.@JsonView"></a>五.@JsonView</h3><p>使用该方法后，代码看起来像这样<br><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">public</span> <span class="keyword">class</span> <span class="title class_">Customer</span> &#123;</span><br><span class="line">    <span class="keyword">public</span> <span class="keyword">interface</span> <span class="title class_">CustomerWithCard</span>&#123;&#125;</span><br><span class="line">    <span class="meta">@JsonView(&#123;CustomerWithCard.class, Card.CardWithCustomer.class&#125;)</span></span><br><span class="line">    <span class="keyword">private</span> Long id;</span><br><span class="line">    <span class="meta">@JsonView(CustomerWithCard.class)</span></span><br><span class="line">    <span class="keyword">private</span> List&lt;Card&gt; cardList;</span><br><span class="line">    <span class="comment">//constructor getter setter...</span></span><br><span class="line">&#125;</span><br><span class="line"><span class="keyword">public</span> <span class="keyword">class</span> <span class="title class_">Card</span> &#123;</span><br><span class="line">    <span class="keyword">public</span> <span class="keyword">interface</span> <span class="title class_">CardWithCustomer</span>&#123;&#125;</span><br><span class="line">    <span class="meta">@JsonView(&#123;Customer.CustomerWithCard.class, CardWithCustomer.class&#125;)</span></span><br><span class="line">    <span class="keyword">private</span> Long id;</span><br><span class="line">    <span class="meta">@JsonView(CardWithCustomer.class)</span></span><br><span class="line">    <span class="keyword">private</span> Customer customer;</span><br><span class="line">    <span class="comment">//constructor getter setter...</span></span><br><span class="line">&#125;</span><br><span class="line"><span class="keyword">public</span> <span class="keyword">class</span> <span class="title class_">CustomerTest</span> &#123;</span><br><span class="line">    <span class="meta">@Test</span></span><br><span class="line">    <span class="keyword">public</span> <span class="keyword">void</span> <span class="title function_">serialize</span><span class="params">()</span> <span class="keyword">throws</span> Exception &#123;</span><br><span class="line">        <span class="type">Customer</span> <span class="variable">customer1</span> <span class="operator">=</span> <span class="keyword">new</span> <span class="title class_">Customer</span>(<span class="number">1L</span>);</span><br><span class="line">        <span class="type">Card</span> <span class="variable">card1</span> <span class="operator">=</span> <span class="keyword">new</span> <span class="title class_">Card</span>(<span class="number">1L</span>, customer1);</span><br><span class="line">        <span class="type">Card</span> <span class="variable">card2</span> <span class="operator">=</span> <span class="keyword">new</span> <span class="title class_">Card</span>(<span class="number">2L</span>, customer1);</span><br><span class="line">        customer1.addCard(card1);</span><br><span class="line">        customer1.addCard(card2);</span><br><span class="line">        List&lt;Card&gt; list = Arrays.asList(card1, card2);</span><br><span class="line">        <span class="type">ObjectMapper</span> <span class="variable">objectMapper</span> <span class="operator">=</span> <span class="keyword">new</span> <span class="title class_">ObjectMapper</span>();</span><br><span class="line">        <span class="comment">//序列化时不包括没有@JsonView注解的属性</span></span><br><span class="line">        objectMapper.configure(MapperFeature.DEFAULT_VIEW_INCLUSION, <span class="literal">false</span>);</span><br><span class="line">        <span class="comment">//序列化需要先指定View</span></span><br><span class="line">        Assertions.assertThat(objectMapper.writerWithView(Customer.CustomerWithCard.class).writeValueAsString(customer))</span><br><span class="line">                .isNotEmpty();</span><br><span class="line">        Assertions.assertThat(objectMapper.writerWithView(Card.CardWithCustomer.class).writeValueAsString(list))</span><br><span class="line">                .isNotEmpty();</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br>下面是序列化的结果<br><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">//serialize customer use CustomerWithCard view</span></span><br><span class="line">&#123;</span><br><span class="line">    <span class="string">&quot;id&quot;</span>: <span class="number">1</span>,</span><br><span class="line">    <span class="string">&quot;cardList&quot;</span>: [</span><br><span class="line">        &#123;</span><br><span class="line">            <span class="string">&quot;id&quot;</span>: <span class="number">1</span></span><br><span class="line">        &#125;,</span><br><span class="line">        &#123;</span><br><span class="line">            <span class="string">&quot;id&quot;</span>: <span class="number">2</span></span><br><span class="line">        &#125;</span><br><span class="line">    ]</span><br><span class="line">&#125;</span><br><span class="line"><span class="comment">//serialize card use CardWithCustomer view</span></span><br><span class="line">[</span><br><span class="line">    &#123;</span><br><span class="line">        <span class="string">&quot;id&quot;</span>: <span class="number">1</span>,</span><br><span class="line">        <span class="string">&quot;customer&quot;</span>: &#123;</span><br><span class="line">            <span class="string">&quot;id&quot;</span>: <span class="number">1</span></span><br><span class="line">        &#125;</span><br><span class="line">    &#125;,</span><br><span class="line">    &#123;</span><br><span class="line">        <span class="string">&quot;id&quot;</span>: <span class="number">2</span>,</span><br><span class="line">        <span class="string">&quot;customer&quot;</span>: &#123;</span><br><span class="line">            <span class="string">&quot;id&quot;</span>: <span class="number">1</span></span><br><span class="line">        &#125;</span><br><span class="line">    &#125;</span><br><span class="line">]</span><br></pre></td></tr></table></figure><br>这种方法的缺点是配置麻烦了一些，但总算勉强能用，下面顺带提一下如何在springmvc(4.3.4版本)中使用该方法，使用了该方法的controller看起来像这样：<br><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">@RestController</span></span><br><span class="line"><span class="meta">@RequestMapping(value = &quot;/api&quot;, produces = MediaType.APPLICATION_JSON_UTF8_VALUE)</span></span><br><span class="line"><span class="keyword">public</span> <span class="keyword">class</span> <span class="title class_">CustomerController</span> &#123;</span><br><span class="line"></span><br><span class="line">    <span class="keyword">private</span> CustomerService customerService;</span><br><span class="line"></span><br><span class="line">    <span class="keyword">public</span> <span class="title function_">CustomerController</span><span class="params">(CustomerService customerService)</span> &#123;</span><br><span class="line">        <span class="built_in">this</span>.customerService = customerService;</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    <span class="meta">@GetMapping(&quot;/customer/&#123;id&#125;&quot;)</span></span><br><span class="line">    <span class="meta">@JsonView(Customer.CustomerWithCard.class)</span></span><br><span class="line">    <span class="keyword">public</span> Customer <span class="title function_">getCustomerById</span><span class="params">(<span class="meta">@PathVariable</span> Long id)</span>&#123;</span><br><span class="line">        <span class="keyword">return</span> customerService.findById(id);</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    <span class="meta">@GetMapping(&quot;/customer/&#123;customerId&#125;/card&quot;)</span></span><br><span class="line">    <span class="meta">@JsonView(Card.CardWithCustomer.class)</span></span><br><span class="line">    <span class="keyword">public</span> List&lt;Card&gt; <span class="title function_">listCardByCustomerId</span><span class="params">(<span class="meta">@PathVariable</span> Long customerId)</span>&#123;</span><br><span class="line">        <span class="keyword">return</span> customerService.listCardByCustomerId(customerId);</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    <span class="meta">@GetMapping(&quot;/customer/&#123;customerId&#125;/card&quot;)</span></span><br><span class="line">    <span class="meta">@JsonView(Card.CardWithCustomer.class)</span></span><br><span class="line">    <span class="keyword">public</span> Page&lt;Card&gt; <span class="title function_">listCardByCustomerIdWithPage</span><span class="params">(<span class="meta">@PathVariable</span> Long customerId, Pageable pageable)</span>&#123;</span><br><span class="line">        <span class="keyword">return</span> customerService.listCardByCustomerIdWithPage(customerId, pageable);</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br>需要注意的是springmvc默认配置MapperFeature.DEFAULT_VIEW_INCLUSION为false，因此如果你在控制器方法上指定了要返回的JsonView，返回的实体类中所有没有该JsonView注解的属性都不会被序列化，这往往正是我们需要的效果，然而也正因为如此，上面代码中带物理分页的listCardByCustomerIdWithPage方法只会返回空对象，因为spring data的PageImpl类中的属性是没有JsonView注解的，我们还需要针对它做一些额外的配置，代码如下:<br><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">/**</span></span><br><span class="line"><span class="comment"> * 用于解决使用json view,并且 MapperFeature.DEFAULT_VIEW_INCLUSION 为false的情况下</span></span><br><span class="line"><span class="comment"> * spring data的page对象无法正确序列化的问题</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@return</span></span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"><span class="meta">@Bean</span></span><br><span class="line"><span class="keyword">public</span> Module <span class="title function_">springDataPageModule</span><span class="params">()</span> &#123;</span><br><span class="line">    <span class="keyword">return</span> <span class="keyword">new</span> <span class="title class_">SimpleModule</span>().addSerializer(Page.class, <span class="keyword">new</span> <span class="title class_">JsonSerializer</span>&lt;Page&gt;() &#123;</span><br><span class="line">        <span class="meta">@Override</span></span><br><span class="line">        <span class="keyword">public</span> <span class="keyword">void</span> <span class="title function_">serialize</span><span class="params">(Page page, JsonGenerator gen, SerializerProvider serializers)</span> <span class="keyword">throws</span> IOException &#123;</span><br><span class="line">            gen.writeStartObject();</span><br><span class="line">            gen.writeFieldName(<span class="string">&quot;size&quot;</span>);</span><br><span class="line">            gen.writeNumber(page.getSize());</span><br><span class="line">            gen.writeFieldName(<span class="string">&quot;number&quot;</span>);</span><br><span class="line">            gen.writeNumber(page.getNumber());</span><br><span class="line">            gen.writeFieldName(<span class="string">&quot;totalElements&quot;</span>);</span><br><span class="line">            gen.writeNumber(page.getTotalElements());</span><br><span class="line">            gen.writeFieldName(<span class="string">&quot;last&quot;</span>);</span><br><span class="line">            gen.writeBoolean(page.isLast());</span><br><span class="line">            gen.writeFieldName(<span class="string">&quot;totalPages&quot;</span>);</span><br><span class="line">            gen.writeNumber(page.getTotalPages());</span><br><span class="line">            gen.writeObjectField(<span class="string">&quot;sort&quot;</span>, page.getSort());</span><br><span class="line">            gen.writeFieldName(<span class="string">&quot;first&quot;</span>);</span><br><span class="line">            gen.writeBoolean(page.isFirst());</span><br><span class="line">            gen.writeFieldName(<span class="string">&quot;numberOfElements&quot;</span>);</span><br><span class="line">            gen.writeNumber(page.getNumberOfElements());</span><br><span class="line">            gen.writeFieldName(<span class="string">&quot;content&quot;</span>);</span><br><span class="line">            serializers.defaultSerializeValue(page.getContent(),gen);</span><br><span class="line">            gen.writeEndObject();</span><br><span class="line">        &#125;</span><br><span class="line">    &#125;);</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure></p>
<h3 id="六-Conclusion"><a href="#六-Conclusion" class="headerlink" title="六.Conclusion"></a>六.Conclusion</h3><p>以上每种方法都有各自的弊端。推荐使用者优先考虑后两种方案。如果我对以上方法的理解存在偏差，或者有人对于该问题有更好的解决方案，欢迎与我讨论。</p>
<p>参考链接：</p>
<ul>
<li><a href="http://www.baeldung.com/jackson-bidirectional-relationships-and-infinite-recursion">Jackson-Bidirectional Relationships</a></li>
<li><a href="https://spring.io/blog/2014/12/02/latest-jackson-integration-improvements-in-spring">Latest Jackson integration improvements in Spring</a></li>
</ul>
]]></content>
      
        <categories>
            
            <category> Backend </category>
            
        </categories>
        
        
        <tags>
            
            <tag> JPA </tag>
            
            <tag> Hibernate </tag>
            
            <tag> Spring </tag>
            
            <tag> Jackson </tag>
            
        </tags>
        
    </entry>
    
    <entry>
      <title><![CDATA[Initial Setup Ubuntu16.04 server]]></title>
      <url>https://blog.staynoob.cn/post/2016/12/Initial-Setup-Ubuntu16-04-server/</url>
      <content type="html"><![CDATA[<blockquote>
<p>这篇文章算是<a href="http://blog.staynoob.cn/post/linux/initial-setup-ubuntu-16.04-desktop/">安装ubuntu16.04后要做的9件事</a>的姊妹篇，区别是上篇文章针对的是ubuntu桌面版，这篇文章针对ubuntu16.04 server版。</p>
</blockquote>
<h3 id="一-配置用户"><a href="#一-配置用户" class="headerlink" title="一.配置用户"></a>一.配置用户</h3><ol>
<li>连接主机<br> 在进行所有操作之前，首先确保你知道主机的外网IP，并且拥有root账户，以上信息一般由各主机提供商提供。在知道以上讯息后可以直接在终端通过以下命令登陆服务器 <figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">ssh root@your_server_ip</span><br></pre></td></tr></table></figure></li>
<li>新增账户<br> linux通用的新增，删除用户命令分别是<code>useradd</code>, <code>userdel</code>，但是ubuntu有一个更方便的新增用户命令<code>adduser</code>，使用<code>adduser</code>命令会以交互式询问的方式来设定用户参数，包括新用户的密码，而使用useradd则需要通过命令行参数的方式来设定选项(如是否需要新建用户目录等)，另外如果使用<code>useradd</code>，还要额外通过<code>passwd</code>命令来设定新用户的密码。这里建议直接使用<code>adduser</code>，以新增用户xy为例，输入 <figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">adduser xy</span><br></pre></td></tr></table></figure>
 除了输入新用户的密码外，该命令还会询问一些额外信息，可以直接按<em>enter</em>键跳过。</li>
<li>添加sudo权限<br> 在ubuntu系统中，将用户添加到sudo用户组会自动获得sudo命令的执行权限，这里可以使用 <figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">usermod -aG sudo xy</span><br></pre></td></tr></table></figure>
 或者 <figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">gpasswd -a xy sudo</span><br></pre></td></tr></table></figure>
 来将xy添加到sudo用户组。至此用户配置就已经完成了，以下列一些其它与用户相关的命令 <figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">passwd xy <span class="comment">#为xy账户设定密码</span></span><br><span class="line">userdel xy <span class="comment">#删除xy账户</span></span><br><span class="line">groupadd <span class="built_in">test</span> <span class="comment">#新增test工作组</span></span><br><span class="line">groupdel <span class="built_in">test</span> <span class="comment">#删除test工作组</span></span><br><span class="line">gpasswd -d xy sudo <span class="comment">#将xy移除sudo用户组</span></span><br></pre></td></tr></table></figure>
</li>
</ol>
<span id="more"></span>
<h3 id="二-配置ssh"><a href="#二-配置ssh" class="headerlink" title="二.配置ssh"></a>二.配置ssh</h3><p>现在已经可以使用xy身份来登陆主机了，但是每次登陆都需要输入密码有些繁琐，可以在自己的常用机器上配置SSH key来跳过这个步骤。</p>
<ol>
<li>生成ssh key<br> 如果你的<em>本地机器</em>已经生成了ssh key可以跳过这个步骤(ssh key通常在用户主目录的.ssh文件夹下)，如果还没有，则可以直接通过如下命令生成 <figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">ssh-keygen</span><br></pre></td></tr></table></figure>
 正常情况下会看到如下输出 <figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">Generating public/private rsa key pair.</span><br><span class="line">Enter file <span class="keyword">in</span> <span class="built_in">which</span> to save the key (/Users/xy/.ssh/id_rsa):</span><br></pre></td></tr></table></figure>
 这里可以指定生成key所存放的路径，一般直接按回车使用默认路径就好了，之后程序会提示你输入“passphrase”，同样可以直接按回车键跳过这一步。如果这一步选择跳过，则私钥可以直接使用，如果你输入了passphrase，则每次使用私钥的时候还需要提供这里输入的passphrase。<br> 完成以上步骤后会在用户目录的.ssh文件夹下看到id_rsa(私钥)与id_rsa.pub(公钥)</li>
<li>ubuntu服务器授权生成的ssh key<br> 首先登陆服务端，进入xy的用户主目录，使用如下命令新建.ssh文件夹 <figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="built_in">mkdir</span> .ssh &amp;&amp; <span class="built_in">chmod</span> 700 .ssh</span><br></pre></td></tr></table></figure>
 再新建authorized_keys文件，将之前生成的id_rsa.pub文件的内容拷贝进去就好了。 <figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">vim ~/.ssh/authorized_keys <span class="comment">#将公钥copy到该文件</span></span><br><span class="line"><span class="built_in">chmod</span> 600 ~/.ssh/authorized_keys</span><br></pre></td></tr></table></figure>
 以上过程多少有些麻烦，如果你的本地机器也是ubuntu16.04，则可以直接使用内建的ssh-copy-id脚本来完成上述工作，具体的命令如下 <figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">ssh-copy-id xy@your_server_ip</span><br></pre></td></tr></table></figure>
</li>
</ol>
<p>至此，每次使用完成该配置的机器登陆ubuntu服务器就不用再提供密码了，顺便提一句，如果本机的用户名与ubuntu服务器上的用户名一致，则用户名也可以省略，登陆命令简化成<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">ssh your_server_ip</span><br></pre></td></tr></table></figure></p>
<h3 id="三-Tips"><a href="#三-Tips" class="headerlink" title="三.Tips"></a>三.Tips</h3><p>ubuntu14.04使用init.d管理服务(daemon)，而ubuntu16.04已经换成了systemd，这里以mysql服务为例简单列一下二者在日常操作上的区别</p>
<ul>
<li>init.d  <figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">sudo /etc/init.d/mysql start</span><br><span class="line">sudo /etc/init.d/mysql restart</span><br><span class="line">sudo /etc/init.d/mysql stop</span><br><span class="line">sudo /etc/init.d/mysql status</span><br><span class="line"><span class="comment">#或者</span></span><br><span class="line">sudo service mysql start   <span class="comment">#启动</span></span><br><span class="line">sudo service mysql restart <span class="comment">#重启</span></span><br><span class="line">sudo service mysql stop    <span class="comment">#停止</span></span><br><span class="line">sudo service mysql status  <span class="comment">#查看状态</span></span><br></pre></td></tr></table></figure></li>
<li>systemd  <figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">systemctl start mysql     <span class="comment">#启动</span></span><br><span class="line">systemctl stop mysql      <span class="comment">#停止</span></span><br><span class="line">systemctl status mysql    <span class="comment">#查看状态</span></span><br><span class="line">systemctl <span class="built_in">enable</span> mysql    <span class="comment">#开机启动</span></span><br><span class="line">journalctl -u mysql       <span class="comment">#查看日志</span></span><br></pre></td></tr></table></figure>
</li>
</ul>
]]></content>
      
        <categories>
            
            <category> System </category>
            
        </categories>
        
        
        <tags>
            
            <tag> Linux </tag>
            
        </tags>
        
    </entry>
    
    <entry>
      <title><![CDATA[魔力数字与快速平方根倒数算法]]></title>
      <url>https://blog.staynoob.cn/post/2016/09/%25E9%25AD%2594%25E5%258A%259B%25E6%2595%25B0%25E5%25AD%2597%25E4%25B8%258E%25E5%25BF%25AB%25E9%2580%259F%25E5%25B9%25B3%25E6%2596%25B9%25E6%25A0%25B9%25E5%2580%2592%25E6%2595%25B0%25E7%25AE%2597%25E6%25B3%2595/</url>
      <content type="html"><![CDATA[<blockquote>
<p>题图来自于世界名模KK的推文，图中的guy&#39;s code就是这篇文章要介绍的算法，它的作用是快速计算平方根倒数，在计算机图形学领域有着广泛的应用。其中&quot;i=0.5f3759df-(i&gt;&gt;1)&quot;是迄今为止我见过的最令我瞠目结舌的一行代码。</p>
</blockquote>
<h3 id="一-来源与背景"><a href="#一-来源与背景" class="headerlink" title="一.来源与背景"></a>一.来源与背景</h3><p>这段代码最早出现在《雷神之锤III竞技场》3D引擎的源码中，2002年左右被人发到了论坛上，并引发了广泛的讨论，下面是代码的原始版本（去掉了c预处理器指令，保留了源码注释）：<br><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line"><span class="type">float</span> <span class="title function_">Q_rsqrt</span><span class="params">( <span class="type">float</span> number )</span></span><br><span class="line">&#123;</span><br><span class="line">	<span class="type">long</span> i;</span><br><span class="line">	<span class="type">float</span> x2, y;</span><br><span class="line">	<span class="type">const</span> <span class="type">float</span> threehalfs = <span class="number">1.5F</span>;</span><br><span class="line"></span><br><span class="line">	x2 = number * <span class="number">0.5F</span>;</span><br><span class="line">	y  = number;</span><br><span class="line">	i  = * ( <span class="type">long</span> * ) &amp;y; <span class="comment">// evil floating point bit level hacking</span></span><br><span class="line">	i  = <span class="number">0x5f3759df</span> - ( i &gt;&gt; <span class="number">1</span> ); <span class="comment">// what the fuck?</span></span><br><span class="line">	y  = * ( <span class="type">float</span> * ) &amp;i;</span><br><span class="line">	y  = y * ( threehalfs - ( x2 * y * y ) ); <span class="comment">// 1st iteration</span></span><br><span class="line"><span class="comment">//	y  = y * ( threehalfs - ( x2 * y * y ) ); // 2nd iteration, this can be removed</span></span><br><span class="line"></span><br><span class="line">	<span class="keyword">return</span> y;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br>经测试这段代码比使用c标准函数(float)(1.0/sqrt(x))快4倍，因而引起了多名学者的注意，不少人发表论文来研究它的工作原理。与它的工作方式一样神秘的是，截止目前为止，仍没有人站出来对这段逼格爆表的代码表示负责。因此0x5f3759df这个魔力数字的得来方式也成了一个谜。<br><span id="more"></span></p>
<h3 id="二-IEEE754浮点数"><a href="#二-IEEE754浮点数" class="headerlink" title="二.IEEE754浮点数"></a>二.IEEE754浮点数</h3><p>要了解它的工作原理，需要先理解浮点数在计算机中的存储方式，大部分编程语言都遵循IEEE754浮点数规范，该规范规定了四种表示浮点数值的方式，其中比较常用的有单精度(32位)，双精度(64位)。c与java使用float指代单精度，double指代双精度，而js默认采用IEEE754双精度来存储数值类型。下表是一个32位浮点数实例：</p>
<table>
<thead>
<tr>
<th style="text-align:center">S(ign,符号)</th>
<th style="text-align:center">E(xponent,指数)</th>
<th style="text-align:center">M(antissa,尾数)</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:center">0</td>
<td style="text-align:center">1111,1111</td>
<td style="text-align:center">010,0000,0000,0000,0000,0000</td>
</tr>
<tr>
<td style="text-align:center">1位</td>
<td style="text-align:center">8位</td>
<td style="text-align:center">23位</td>
</tr>
</tbody>
</table>
<p>下面简单介绍它们的存储格式<br>1.符号位(sign)<br>    符号位使用1来代表负数，0代表正数。<br>2.指数位(exponent)<br>    指数可以是正指数或是负指数，但其表示方式并不遵循传统的二进制补码规则，而是先将其视为无符号数，再让指数偏移固定值来代表实际值，以单精度浮点数来举例，IEEE754标准规定偏移值为127($2^{b-1}-1$，b是指数的位数，此例是8)，因此指数部分需要减去127从而得到实际值。比如上例中指数部分为11111111，转换成10进制就是255，再减去127得到128，这里128就是它的实际指数。再比如00000000表示的就是-127，按照这个规则，它能够表示-127~128范围内的数字。（假设采用二进制补码规则，11111111表示的是-1，能够表示的数值范围是-128~127）<br>3.尾数位(mantissa)<br>    IEEE754中的尾数又称为有效数(significant)，它用来表示科学计数法的小数部分，同时它规定第一个有效数必定是1，因此1就不需要存储，例如单精度浮点数有23位用于存储有效数，加上最左边没有存储的1，一共就有24位。继续拿上表的例子来说，可以把它的尾数左边四位看成1010，因此它的有效数实际值就是$(2^{23}+2^{21})\times 2^{-23}=1+2^{21}\times 2^{-23}$。<br>最后，我们可以得到上表中的浮点数实际值为：$$(-1)^s\times (1+m\times 2^{-23})\times 2^{e-127}=(-1)^0\times(1+2^{21}\times 2^{-23})\times2^{255-127}=1.25\times 2^{128}$$</p>
<h3 id="三-一行神奇的代码"><a href="#三-一行神奇的代码" class="headerlink" title="三.一行神奇的代码"></a>三.一行神奇的代码</h3><p>下面来分析&quot;i=0x5f3759df-(i&gt;&gt;1)&quot;这行充满魔力的代码，它的作用是获取一个平方根倒数的近似值。其工作原理证明过程非常的复杂(参考论文<a href="http://www.lomont.org/Math/Papers/2003/InvSqrt.pdf">Fast Inverse Square Root</a>)，这里提供wikipedia上的证明版本（尽管这是我个人认为最简单的证明版本，但看起来依然惨不忍睹，不求甚解的读者可以跳过证明部分）：<br>我们要计算的值是$\frac{1}{\sqrt{x}}$，令$y=\frac{1}{\sqrt{x}}=x^{-\frac{1}{2}}$，等式两边同时取对数可以得到$logy=log(x^{-\frac{1}{2}})=-\frac{1}{2}logx$。<br>现在假设输入值x是一个n位的IEEE754浮点数，由于要求输入是正数，因此x的符号位一定是0，假设它有i个指数位，尾数位就是n-i-1。设它的指数位为$E_x$，表示的指数是$e_x$，尾数位为$M_x$，表示的有效数字为$1+m_x$，根据上面的对浮点数表示方式的定义，可以得出：$$e_x=E_x-B$$$$m_x=\frac{M_x}{L}$$<br>其中$B=(2^i-1)$，$L=2^{n-i-1}$，此时输入的x的值为$(1+m_x)\times 2^{e_x}$，同理y值也可以表示成$(1+m_y)\times 2^{e_y}$，将它们代入上面的对数方程可以得到：$$log((1+m_y)\times 2^{e_y})=-\frac{1}{2}log((1+m_x)\times 2^{e_x})$$$$\Rightarrow log(1+m_y)+e_y=-\frac{1}{2}log(1+m_x)-\frac{1}{2}e_x$$现在假设我们用$\sigma_a$来表示$log(1+m_y)$与$m_y$之间的误差，$\sigma_b$表示$log(1+m_x)$与$m_x$之间的误差，即$\sigma_a=log(1+m_y)-m_y$，$\sigma_b=log(1+m_x)-m_x$，将它们代入上式得到：$$m_y+\sigma_a+e_y=-\frac{1}{2}(m_x+\sigma_b)-\frac{1}{2}e_x$$$$\Rightarrow \frac{M_y}{L}+E_y-B+\sigma_a=-\frac{1}{2}(\frac{M_x}{L}+\sigma_b)-\frac{1}{2}(E_x-B)$$$$\Rightarrow M_y+E_yL=\frac{3}{2}BL-(\sigma_a+\frac{\sigma_b}{2})L-\frac{1}{2}(M_x+E_xL)$$<br>见证奇迹的时刻到了，回顾一下这张表：</p>
<table>
<thead>
<tr>
<th style="text-align:center">S(ign,符号)</th>
<th style="text-align:center">E(xponent,指数)</th>
<th style="text-align:center">M(antissa,尾数)</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:center">1位</td>
<td style="text-align:center">i位</td>
<td style="text-align:center">n-i-1位</td>
</tr>
</tbody>
</table>
<p>由于$L=2^{n-i-1}$，因此$M_x+E_xL$，$M_y+E_yL$正好是x与y所代表的整数值，如果我们用R来代表原方程中$\frac{3}{2}BL-(\sigma_a+\frac{\sigma_b}{2})L$这一部分，会得到$I_y=R-\frac{I_x}{2}$，将除以2替换成计算机更喜欢的右移位后，就成了这行神奇的代码。现在只剩下最后一个问题了，R值是多少？</p>
<h3 id="四-魔力数字"><a href="#四-魔力数字" class="headerlink" title="四.魔力数字"></a>四.魔力数字</h3><p>根据上面段落的推理过程，可以知道魔力数字R应该等于$\frac{3}{2}BL-(\sigma_a+\frac{\sigma_b}{2})L$，其中B与L都是常量，如果是32位浮点数，则B等于127，L等于$2^{23}$。而$\sigma_a=log(1+m_y)-m_y$，$\sigma_b=log(1+m_x)-m_x$。现在可以构建一个函数f(x)=log(1+x)-x，由于$m_x$，$m_y$均为尾数部分，因此自变量x的取值范围是[0,1]，到这里我的想法是利用积分计算出函数f(x)=log(1+x)-x在[0,1]区间的平均值，从而得出$\sigma$的近似值，即：<br>$$\int_0^1\left[log(1+x)-x\right]dx$$如果我没算错的话，结果应该是$2-\frac{1}{ln2}-\frac{1}{2}\approx0.05730495911$。(wikipedia对$\sigma$的计算结果是0.0450461875...，我没看懂它的计算方式，如果有数学高手看到这里希望能指导一下)。最后将该值带回原式：<br>$$1.5\times2^{23}\times(127 - 1.5\times0.05730495911)\approx1597308761=0x5f34ff59$$同样这里我的计算结果与源码(0x5f3759df)不同，我也不知道错在哪里，希望有大神指导。</p>
<h3 id="五-牛顿法"><a href="#五-牛顿法" class="headerlink" title="五.牛顿法"></a>五.牛顿法</h3><p>至于代码中<br><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">	y  = y * ( threehalfs - ( x2 * y * y ) ); <span class="comment">// 1st iteration</span></span><br><span class="line"><span class="comment">//	y  = y * ( threehalfs - ( x2 * y * y ) ); // 2nd iteration, this can be removed</span></span><br></pre></td></tr></table></figure><br>这一部分则是使用牛顿迭代法来获取更精确的计算结果，关于牛顿法的详情可以参见我的上一篇文章<a href="http://blog.staynoob.cn/post/algorithm/integer-arithmetic-karatsuba-multiplication/#二-牛顿法（Newton-Raphson-method）">牛顿迭代法</a>，这里不再赘述。</p>
<h3 id="六-总结"><a href="#六-总结" class="headerlink" title="六.总结"></a>六.总结</h3><p>关于这段代码的出处至今仍有争议，相传是由天才程序员<a href="https://zh.wikipedia.org/wiki/%E7%B4%84%E7%BF%B0%C2%B7%E5%8D%A1%E9%A6%AC%E5%85%8B">约翰·卡马克</a>所开发（约翰·卡马克是编程领域畅销书<a href="https://book.douban.com/subject/1152971/">DOOM启示录</a>的主角），然而不管这段代码出自于谁，我都觉得是码农届的荣耀，毫不夸张的说我因为0x5f3759df这个数字在家花了一个月时间重温高数，还有里面的&quot;evil floating point bit level hacking&quot;，深刻的让我觉得这段代码的作者是用灵魂在撸代码。</p>
<p>参考链接：</p>
<ul>
<li><a href="https://zh.wikipedia.org/wiki/%E5%B9%B3%E6%96%B9%E6%A0%B9%E5%80%92%E6%95%B0%E9%80%9F%E7%AE%97%E6%B3%95">平方根倒数速算法</a></li>
<li><a href="http://www.lomont.org/Math/Papers/2003/InvSqrt.pdf">Fast Inverse Square Root</a></li>
</ul>
]]></content>
      
        <categories>
            
            <category> Algorithm </category>
            
        </categories>
        
        
    </entry>
    
    <entry>
      <title><![CDATA[数值运算与牛顿迭代法]]></title>
      <url>https://blog.staynoob.cn/post/2016/06/%25E6%2595%25B0%25E5%2580%25BC%25E8%25BF%2590%25E7%25AE%2597%25E4%25B8%258E%25E7%2589%259B%25E9%25A1%25BF%25E8%25BF%25AD%25E4%25BB%25A3%25E6%25B3%2595/</url>
      <content type="html"><![CDATA[<blockquote>
<p>一直不明白计算机如何通过软件实现乘法，除法，开方等常见的数值运算，最近我在mit开源课程中得到了答案，以下是学习心得。</p>
</blockquote>
<h3 id="一-乘法运算"><a href="#一-乘法运算" class="headerlink" title="一.乘法运算"></a>一.乘法运算</h3><h4 id="1-暴力算法"><a href="#1-暴力算法" class="headerlink" title="1.暴力算法"></a>1.暴力算法</h4><p>假设有两个十进制数x=12345678，y=87654321，我们需要计算x与y的乘积，可以先采用分治(Divide and Conquer)的思想，将每个数字拆分成两半，如下表所示：</p>
<table>
<thead>
<tr>
<th style="text-align:center">1234</th>
<th style="text-align:center">5678</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:center">$x_0$</td>
<td style="text-align:center">$x_1$</td>
</tr>
</tbody>
</table>
<table>
<thead>
<tr>
<th style="text-align:center">8765</th>
<th style="text-align:center">4321</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:center">$y_0$</td>
<td style="text-align:center">$y_1$</td>
</tr>
</tbody>
</table>
<p>很容易得到：$$x=10^{d/2}x_0+x_1=12340000+5678$$$$y=10^{d/2}y_0+y_1=87650000+4321$$$$\Rightarrow xy=(10^{d/2}x_0+x_1)(10^{d/2}y_0+y_1)$$$$\Rightarrow xy=10^dx_0y_0+10^{d/2}(x_0y_1+x_1y_0)+x_1y_1$$这样下来每一次递归，都可以把两个数的乘积，转换成四组更小精度数的乘积之和，重复这个过程直到把问题划分成多个个位数的乘积之和，这个时候简单的使用九九乘法表就能得到每一项的结果。整个过程其实与我们小学学的计算方式很类似，你可以在草稿纸上尝试计算12*34来模拟这个过程。<br>当然，以上是人类的思维方式，把上面的数字转换成相应的二进制数就能模拟计算机的思维方式，唯一的不同是计算机的个位数相乘不需要乘法口诀，进行一次位与(&amp;)运算就可以了。下面是它的时间复杂度（计算过程与一般分治算法复杂度计算过程类似，下文不再提及）：$$T(N)=4T(N/2)+\Theta(N)=\Theta(N^2)$$<br><span id="more"></span></p>
<h4 id="2-Karatsuba算法"><a href="#2-Karatsuba算法" class="headerlink" title="2.Karatsuba算法"></a>2.Karatsuba算法</h4><p>上面的暴力算法将x乘以y拆分成了$x_0y_0,x_0y_1,x_1y_0,x_1y_1$四个更小规模的子问题，而一名叫Karatsuba的学者发现，在已知$x_0y_0,x_1y_1$的情况下，可以使用$(x_0+x_1)(y_0+y_1)-x_0y_0-x_1y_1$来得到$x_0y_1+x_1y_0$的值，整个算式变成了：$$xy=10^dx_0y_0+10^{d/2}[(x_0+x_1)(y_0+y_1)-x_0y_0-x_1y_1]+x_1y_1$$这样每次递归调用只需要进行三次乘法运算，此时复杂度为：$$T(N)=3(T/2)+\Theta(N)=\Theta(N^{\log_23})\approx\Theta(N^{1.58})$$在Karatsuba之后，Toom-Cook通过每次将数字切分成多个部分进一步提升了算法的性能，例如将前面的x切分成$x_0,x_1,x_2$三个部分，此时算法的复杂度为：$$T(N)=5(T/3)+\Theta(N)=\Theta(N^{\log_35})\approx\Theta(N^{1.46})$$Schönhage–Strassen通过快速傅里叶转换将复杂度提升至$\Theta(NlogNloglogN)$，但这些显然只有在精度需求非常大的情况下才有意义。大部分情况下使用暴力算法或Karatsuba算法就已经足够了。</p>
<h3 id="二-牛顿法（Newton-Raphson-method）"><a href="#二-牛顿法（Newton-Raphson-method）" class="headerlink" title="二.牛顿法（Newton-Raphson method）"></a>二.牛顿法（Newton-Raphson method）</h3><p>牛顿法是一种求解方程近似根的方法，他在计算机领域应用非常广泛，下文要提到的除法运算与开方运算都需要通过它实现。下面是方法的简介：<br>假设现在我们需要计算函数f(x)的根(当f(x)取0时，x的值称作方程的根)<br>1.先取一个大概的值$x_0$，代入该函数可以得到$y_0$<br>2.过$(x_0,y_0)$作一条切线，该切线的斜率为$f&#39;(x_0)$（这里$f&#39;$表示函数$f$的导数）<br>3.计算该切线与x轴的焦点$x_1$，显然这里有：$$k=\frac{y_0}{x_0-x_1}=f&#39;(x_0)=\frac{f(x_0)}{x_0-x_1}$$$$\Rightarrow x_1=x_0-\frac{f(x_0)}{f&#39;(x_0)}$$<br>4.重复上面的过程，直到得到令你满意的精度为止。wikipedia上的这张图清晰的展示了它的迭代过程:</p>
<p><img src="/img/content/newton_lteration.gif" alt="newton_lteration.gif"></p>
<h3 id="三-平方根算法"><a href="#三-平方根算法" class="headerlink" title="三.平方根算法"></a>三.平方根算法</h3><p>牛顿法的一个常见的应用场景是计算算术平方根，现在假设我们需要计算$\sqrt{a}$的值，这等价于计算$f(x)=x^2-a$的根，将其代入上面的公式可得：$$x_1=x_0-\frac{f(x_0)}{f&#39;(x_0)}=x_0-\frac{x^2-a}{2x}=\frac{x_0+a/x_0}{2}$$用java代码来简单模拟求解过程：<br><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">public</span> <span class="keyword">static</span> <span class="type">double</span> <span class="title function_">sqrt</span><span class="params">(<span class="type">double</span> input)</span>&#123;</span><br><span class="line">    <span class="type">double</span> x=input;</span><br><span class="line">    <span class="keyword">for</span> (<span class="type">int</span> <span class="variable">i</span> <span class="operator">=</span> <span class="number">0</span>; i &lt; <span class="number">5</span>; i++) &#123;</span><br><span class="line">        x=(x+input/x)&gt;&gt;<span class="number">1</span>;</span><br><span class="line">        System.out.println(i+<span class="string">&quot;次迭代:&quot;</span>+x);</span><br><span class="line">    &#125;</span><br><span class="line">    <span class="keyword">return</span> x;</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">void</span> <span class="title function_">main</span><span class="params">(String... args)</span>&#123;</span><br><span class="line">	sqrt(<span class="number">2d</span>);</span><br><span class="line">    <span class="comment">/*</span></span><br><span class="line"><span class="comment">    参考值 :1.414213562373095</span></span><br><span class="line"><span class="comment">    程序输出:</span></span><br><span class="line"><span class="comment">    0次迭代:1.5</span></span><br><span class="line"><span class="comment">    1次迭代:1.4166666666666665</span></span><br><span class="line"><span class="comment">    2次迭代:1.4142156862745097</span></span><br><span class="line"><span class="comment">    3次迭代:1.4142135623746899</span></span><br><span class="line"><span class="comment">    4次迭代:1.414213562373095</span></span><br><span class="line"><span class="comment">    */</span></span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br>大部分情况下，牛顿法具有平方级的收敛性能，即每迭代一次，结果的有效数字将增加一倍，这意味着只需要logN次迭代就能得到N精度的结果。但我们不能说它的复杂度是对数级的，因为我们忽视$a/x_0$这项操作，要知道以上算法的复杂度，必须先来看看除法运算的复杂度。</p>
<h3 id="四-除法运算"><a href="#四-除法运算" class="headerlink" title="四.除法运算"></a>四.除法运算</h3><p>假设我们需要计算a/b，我们可以先将a/b转换成$a\frac{1}{b}$，由于乘法操作是已知的，因此这里只需要计算1/b就可以了，接下来可以构建一个函数$f(x)=\frac{1}{x}-b$，然后继续用牛顿法求它的根：$$x_1=x_0-\frac{f(x_0)}{f(x_1)}=x_0-\frac{1/x_0-b}{-1/x_0^2}=x_0(2-bx_0)$$这样就把除法运算全部转换成了乘法运算，同样我们需要logN次迭代才能得出N精度的结果，这里假设乘法运算的复杂度是$N^\alpha$，于是可以简单的得出复杂度的上界是$N^\alpha logN$，但仔细观察会发现，并不是每一次迭代都需要进行N精度的乘法，更准确的复杂度推导如下：$$T(N)=1^\alpha+2^\alpha+4^\alpha+8^\alpha+\cdots+N^\alpha$$$$\Rightarrow T(N)=(2^\alpha)^0+(2^\alpha)^1+(2^\alpha)^2+\cdots+(2^\alpha)^{logN}$$$$\Rightarrow 2^\alpha T(N)=(2^\alpha)^1+(2^\alpha)^2+\cdots+(2^\alpha)^{logN}+(2^\alpha)^{logN+1}$$$$\Rightarrow (2^\alpha-1)T(N)=(2^\alpha)^{logN+1}-(2^\alpha)^0$$$$\Rightarrow T(N)=\frac{(2N)^\alpha-1}{2^\alpha-1}=\Theta(N^\alpha)$$最终得出结论，使用牛顿法进行除法运算其复杂度与所用的乘法算法复杂度一致，类似的，容易证明使用牛顿法计算平方根的时间复杂度也是$\Theta(N^\alpha)$。<br>以上是这篇文章的全部内容，下篇文章来介绍一个神奇的平方根倒数算法，其中同样涉及到牛顿法。</p>
]]></content>
      
        <categories>
            
            <category> Algorithm </category>
            
        </categories>
        
        
    </entry>
    
    <entry>
      <title><![CDATA[香农信息论与回答老鼠喝药问题的正确姿势]]></title>
      <url>https://blog.staynoob.cn/post/2016/05/%25E9%25A6%2599%25E5%2586%259C%25E4%25BF%25A1%25E6%2581%25AF%25E8%25AE%25BA%25E4%25B8%258E%25E5%259B%259E%25E7%25AD%2594%25E8%2580%2581%25E9%25BC%25A0%25E5%2596%259D%25E8%258D%25AF%25E9%2597%25AE%25E9%25A2%2598%25E7%259A%2584%25E6%25AD%25A3%25E7%25A1%25AE%25E5%25A7%25BF%25E5%258A%25BF/</url>
      <content type="html"><![CDATA[<blockquote>
<p>最近在学习算法的过程中频繁接触到信息论，有碍于自己的数学水平，无力窥其全貌。但仅仅是一个入门级的了解便足以让我为之倾倒。这篇文章简单介绍信息论，之后来看看如何利用它秒解一些智力题。</p>
</blockquote>
<h3 id="一-老鼠喝药问题"><a href="#一-老鼠喝药问题" class="headerlink" title="一.老鼠喝药问题"></a>一.老鼠喝药问题</h3><p>先来看看要解决的问题:</p>
<blockquote>
<p>有 1000 个一模一样的瓶子，其中有 999 瓶是普通的水，有一瓶是毒药。你有一些小白鼠用来找出毒药，喝下毒药的老鼠将会在第二天死亡，最少需要多少只小白鼠才能保证在第二天找出哪一瓶是毒药？</p>
</blockquote>
<p>你可以带着这个问题阅读接下来的文章，如果你善于思考，有可能在中途就已经得到答案。<br><span id="more"></span></p>
<h3 id="二-信息的量化与信息熵-读音为shang-英文是entropy"><a href="#二-信息的量化与信息熵-读音为shang-英文是entropy" class="headerlink" title="二.信息的量化与信息熵(读音为shang,英文是entropy)"></a>二.信息的量化与信息熵(读音为shang,英文是entropy)</h3><p>首先不要被这个标题所吓倒，它的概念非常的简洁。这里引用吴军博士在数学之美系列文章中所举的例子来介绍这个概念。</p>
<blockquote>
<p>假设你错过了2014年的世界杯，你可以问一个知道比赛结果的观众，但他只回答是或否。你最多需要问多少个问题才能知道哪支球队是冠军？<br>你可以先对所有球队进行编号，先问“冠军球队在1-16号中吗？”如果他回答是，就继续问“冠军在1-8号中吗？”整个过程类似于二分查找，这样最多需要问5次就能得到结果。因此“谁是世界杯冠军”这条信息的信息量就是5比特(bit)，如果某一年有64支球队进入决赛阶段，那么你需要多问一次，此时该条信息的信息量变成6比特。至此你可以发现信息量的大小与所有可能情况的对数函数有关(log32=5，log64=6，注：本文所有对数以2为底)</p>
</blockquote>
<p>懂球的读者会发现可能不需要5次就能猜出谁是冠军，因为像巴西，西班牙，意大利这样的球队夺冠的概率比某些亚洲国家要大，因此可以把少数几支夺冠热门球队分成一组，其它球队分成另一组，然后猜冠军球队是否在这几支热门球队中。重复这个过程。也许三次或者四次就能得到结果。因此当每支球队夺冠概率不等时，“谁是世界杯冠军”的信息量小于5比特。<br>香农在他著名的论文“<a href="https://en.wikipedia.org/wiki/A_Mathematical_Theory_of_Communication">通信的数学原理(A Mathematical Theory of Communication)</a>”中指出，它准确的信息量应该通过如下方式计算：$$ H=-(p_1\log p_1+p_2\log p_2+\cdots +p_{32}\log p_{32}) $$其中$p_1,p_2,\ldots ,p_{32}$分别代表每支球队夺冠的概率，H则被香农定义为“信息熵”，它的单位是比特。这里使用“熵”这个字是因为它与热力学中的熵有一定联系，信息熵用于表示信源随机系统的不确定程度。而热力学的“熵”用于表示系统中无序(无法再利用)的能量。<br>如果以上看不懂也没关系，当所有球队夺冠概率相等时，p=1/32，以上公式可以简化成：$$H=-\log (\frac{1}{32})=5(bit)$$你可以将“谁是世界杯冠军”这条信息的信息量大小理解成它的不确定性，越是不确定的事件则需要越大的信息量来将它确定。比如说彩票双色球的中奖号码，我没买过彩票，有兴趣的读者可以自己计算一下。再比如有人告诉你明天太阳从东边出来(假设其概率为100%)。套用公式计算其信息量就是：$$H=-\log (1)=0(bit)$$于是我们严格的证明了它是一句废话。</p>
<h3 id="三-条件熵-Conditional-Entropy-与互信息-Mutual-Information"><a href="#三-条件熵-Conditional-Entropy-与互信息-Mutual-Information" class="headerlink" title="三.条件熵(Conditional Entropy)与互信息(Mutual Information)"></a>三.条件熵(Conditional Entropy)与互信息(Mutual Information)</h3><p>这进行这一节的叙述之前，先来了解一些数学知识，由于我大概只有初中毕业的数学水平，因此你应该有自信能够理解它们，如果你感觉到难以理解，很有可能是因为我的表达有问题，建议你继续在网上阅读一些其它的资料。</p>
<ul>
<li><a href="https://zh.wikipedia.org/wiki/%E6%9D%A1%E4%BB%B6%E6%A6%82%E7%8E%87%E5%88%86%E5%B8%83">条件概率分布(Conditional Probability)</a><br>  条件概率分布描述了已知第二个随机变量Y的前提下，X的概率分布，用P(x|y)表示。比如我用P(x)表示世界杯足球赛1号队伍夺冠的概率，P(y)表示冠军在1-16号队伍之间的概率，假设所有球队夺冠概率相等，可以得到P(x)=1/32，P(y)=1/2，P(x|y)=1/16。用语言描述就是1号队伍夺冠的概率为1/32，但是如果已知冠军在1-16号队伍之中，则1号队伍夺冠的概率变为1/16。</li>
<li><a href="https://zh.wikipedia.org/wiki/%E8%81%94%E5%90%88%E5%88%86%E5%B8%83">联合概率分布(JointProbability))</a><br>  假设存在两个随机变量X和Y，则我们将X=x并且Y=y的概率称为联合概率分布，用P(x,y)表示。结合上面的概念可以得到：$$P(x,y)=P(x)\times P(y|x)=P(y)\times P(x|y)$$用语言来描述就是x,y两件事同时发生的概率等于x单独发生的概率乘以在已经发生x的前提下，y发生的概率。继续用上面的例子来看，可以得到：$$P(x,y)=P(y)\times P(x|y)=1/32$$即世界杯1号队伍夺冠(x)与冠军在1-16号之间(y)这两件事同时发生的概率为1/32(别忘了假设所有球队夺冠概率相等)。</li>
</ul>
<p>如果理解了上面的数学概念，后面的内容就非常容易解释了，世界杯32支球队中谁是冠军的不确定性(信息量)是5bit，我们必须通过引入外部信息（即提问）来消除这个不确定性，提问的过程可以看做是降低它的不确定性的过程，但我们具体应该提什么样的问题呢？我们可以直接了当的问谁是冠军，当得到回答后事件的不确定性直接降为0，也可以问诸如“今天天气怎么样？”，“你吃过饭了吗？”之类的不相干的问题，得到回答后它的不确定性仍然是5bit。此时如何来量化我们提出的问题与事件本身的相关性？这就需要引入条件熵与互信息的概念。</p>
<ul>
<li><a href="https://zh.wikipedia.org/wiki/%E6%9D%A1%E4%BB%B6%E7%86%B5">条件熵(Conditional Entropy)</a><br>  条件熵描述了在已知第二个随机变量Y的前提下，随机变量X的不确定性还剩多少，用H(X|Y)表示，计算公式为：$$H(X|Y)=-\sum_{x\in\mathcal X, y\in\mathcal Y}p(x,y)\log\,p(x|y)$$假如我们用X来表示“谁是世界杯冠军”，用Y来表示“世界杯冠军是否在1-16号球队之中”，那么综合我们学到的内容可以得到p(x)=1/32，p(y)=1/2，p(x,y)=1/32，p(x|y)=1/16。套用公式后得到H(X|Y)=4bit。这意味着假如我们知道了“世界杯冠军是否在1-16号球队之中”，那么“谁是世界杯冠军”这件事的不确定性就只剩4bit了。</li>
<li><a href="https://zh.wikipedia.org/wiki/%E4%BA%92%E4%BF%A1%E6%81%AF">互信息(Mutual Information)</a><br>  互信息量化了两个随机变量之间的相关性，它的定义其实与条件熵很类似，它表示在已知第二个随机变量Y的前提下，随机变量X的不确定性应该减去多少。用I(X;Y)来表示。根据定义很容易得到：$$I(X;Y)=H(X)-H(X|Y)=H(Y)-H(Y|X)$$同样用上面的例子。已知H(X)=5bit，H(X|Y)=4bit，因此I(X;Y)=1bit。这意味着“世界杯冠军是否在1-16号球队之中”与“谁是世界杯冠军”这两个问题之间的相关性是1bit，你可以自己计算一下“今天天气怎么样？”这个问题与“谁是世界杯冠军”的相关性。</li>
</ul>
<p>如果你感觉已经理解了上面的内容，下面咱们不妨来做一个小测验，这是一道经典的智力题。</p>
<blockquote>
<p>假设有27个小球，其中一个小球质量高于其它小球，你有一个天平，请问要找出质量较高的小球。最少需要称多少次？</p>
</blockquote>
<p>我们可以用X代表“27个小球哪一个质量比其它小球大？”，用Y表示用天平进行一次称重，容易计算出X的信息量H(X)=log27(bit)，Y的信息量H(Y)=log3(bit)(称重的结果有大于，小于，等于)，因此互信息I(X;Y)的最大值也是log3，使用H(X)除以I(X;Y)可以得出理论上最少需要称量的次数，即log27/log3。依据对数的性质，该式等于$\log _327$，最少需要称3次。</p>
<h3 id="四-信息论的应用场景"><a href="#四-信息论的应用场景" class="headerlink" title="四.信息论的应用场景"></a>四.信息论的应用场景</h3><p>信息论目前广泛应用于物理学，统计学，密码学等领域，包括今天的整个移动通讯与互联网都是建立在它的理论基础之上的。然而对于以上领域我几乎一窍不通，下面列举一些它在码农领域的作用。</p>
<ul>
<li>数据压缩，编码，解码<br>  信息论直接决定了无损压缩的极限。假设你现在要将“谁是世界杯冠军”这条信息压缩并通过网络传输，那么它的极限是5bit。低于5bit则无法涵盖所有可能性(具体的传输方式是发送端与接受端先约定好编码规则，比如用00001表示巴西获胜，00010表示意大利获胜，根据二进制规则，5位二进制数最大能表达32种可能)。人类之间的沟通也可以看做是编码-传输-解码的过程，即我将我想表达的内容通过一种语言进行编码，使用一些媒体介质传输给听众，听众再通过相同的语言进行解码。好的作家往往具备高超的编码技巧，使用较少文字就能准确表达意图，带给读者身临其境的感觉，当然这同时也要求读者具有一定的解码技巧，否则数据在这次传输过程中就会失真。因此高压缩率的语言虽然便于传输与存储，但它的缺点是容易受到噪声干扰，编码与解码的运算量比较大，比如说文言文。</li>
<li>弱人工智能<br>  这里我暂且把基于概率与统计而形成的人工智能称为弱人工智能，比如说机器翻译，其实机器并不能理解它所翻译的内容，它是一个根据海量的数据模型与上下文相关词汇来降低语义的不确定性的过程。比如说football，计算机可以选择将它翻译成足球或者橄榄球，但如果在上下文中带有Maradona(马拉多纳)，则大大降低了这个不确定性。（计算机又是如何认识Maradona的呢？它先从大量文本中找出与足球一起出现的互信息较高的词。更详细的内容还是推荐阅读《数学之美》）。类似的领域还有语音识别，聊天机器人，搜索引擎关键字分析等等。</li>
<li>算法分析<br>  在<a href="http://blog.staynoob.cn/post/algorithm/insertion-shell-merge-quick-sort-algorithm-2/#二-最快的排序算法">聊聊4种主流排序算法</a>这篇文章中曾经提到如何证明基于比较的排序算法复杂度不可能优于$\theta(NlogN)$。证明过程使用了二叉决策树，而使用信息论的证明更加简洁明了，一个大小为N，所有元素都不相等的数组，其信息量为log(N!)，而每一次比较的互信息是1bit，因此至少需要log(N!)(约等于NlogN)次比较。如果你也是码农的话，这方面应用场景应该也不用我多做解释了。我也正是因为在学习算法的过程中才开始了解信息论的。</li>
</ul>
<p>如果以上还不足以说服你，或者说你对上面这些领域都不感冒，我再说一个你可能感兴趣的。<br>上世纪90年代初，改进最大熵模型训练算法的Della Pietra兄弟退出学术届，和一些当时在IBM做语言识别的同事一起加入<a href="http://wiki.mbalib.com/wiki/%E8%A9%B9%E5%A7%86%E6%96%AF%C2%B7%E8%A5%BF%E8%92%99%E6%96%AF">詹姆斯·西蒙斯</a>领导的文艺复兴公司(Renaissance Technologies Corp.)。利用最大熵模型和一些其它数学理论对股票进行预测。1988-2008二十年里，该公司的大奖章(Medallion)对冲基金年均回报率高达34%，文艺复兴公司现在也成了世界上最成功的对冲基金公司。</p>
<h3 id="五-正确姿势"><a href="#五-正确姿势" class="headerlink" title="五.正确姿势"></a>五.正确姿势</h3><p>回到我们要解决的问题，1000个瓶子中哪一瓶是毒药的信息量是log1000，向上取整得到10bit，一支老鼠的生死正好能表示1bit。最终得出答案10支老鼠。具体的操作流程在<a href="https://www.zhihu.com/question/19676641">知乎：老鼠喝药问题</a>中已经有无数大神进行解答了，本文也就不再赘述。<br>最后再说一些题外话，这个问题最开始是在与同事聊天过程中被问到的，我经过一晚上的冥思苦想，最终没能得到答案，第二天同事在告知正确解法后随口又问了一句：“假设把时限放宽到两天，那么至少需要多少支老鼠？”于是又是一晚上的冥思苦想……<br>智商上的缺陷常常需要后天努力来弥补，在学习了信息论之后，类似的问题无论如何变化，都很容易套用公式计算出理论最优解。</p>
<p>参考链接：</p>
<ul>
<li><a href="https://www.zhihu.com/question/27068465">知乎：香农的信息论究竟牛在哪里？</a></li>
<li><a href="http://googlechinablog.blogspot.com/?q=%E6%95%B0%E5%AD%A6%E4%B9%8B%E7%BE%8E">Google黑板报：数学之美系列</a></li>
</ul>
]]></content>
      
        <categories>
            
            <category> Mathematic </category>
            
        </categories>
        
        
    </entry>
    
    <entry>
      <title><![CDATA[CSS清除浮动的原理与最佳实践]]></title>
      <url>https://blog.staynoob.cn/post/2016/05/CSS%25E6%25B8%2585%25E9%2599%25A4%25E6%25B5%25AE%25E5%258A%25A8%25E7%259A%2584%25E5%258E%259F%25E7%2590%2586%25E4%25B8%258E%25E6%259C%2580%25E4%25BD%25B3%25E5%25AE%259E%25E8%25B7%25B5/</url>
      <content type="html"><![CDATA[<blockquote>
<p>网上关于CSS清除浮动的文章围起来简直可以绕地球两圈。原因是大多数码农总会有一种错觉，总感觉自己写的代码比别人的更加清晰易懂，一言不合就重构，我也一样。</p>
</blockquote>
<h3 id="一-为什么要清除浮动"><a href="#一-为什么要清除浮动" class="headerlink" title="一.为什么要清除浮动"></a>一.为什么要清除浮动</h3><p>浮动会造成元素脱离普通流，此时普通流中的元素会当做浮动的元素完全不存在一样，这会造成一些布局上的问题，其一是普通流中的元素会被浮动的元素覆盖(文本内容不会被覆盖)，其二是它的容器不再会根据它的高度来调整自身的高度，下面的例子演示了这两个问题：<br>
<div class="clearfix" style="border:5px solid #a57958;padding:10px">
	<p>按下按钮查看清除浮动对布局的影响</p>
    <button id="before" class="btn btn-info">清除浮动前</button>
    <button id="after" class="btn btn-info">清除浮动后</button>
    <div style="border:5px solid #a57958;margin-top:10px;">
        我是容器
        <div style="color:black;background-color:wheat;padding:10px;width:30%;height:150px;float:left;">div1
        我浮动了</div>
        <div id="demo" style="background-color:skyblue;padding:10px;">div2
        我是普通流中的块级元素
        我的宽度应该是100%
        我的文字没有被覆盖</div>
    </div>
</div>
<script>
	$("#before").on("click",function(){
    	$("#demo").css("clear","none");
    });
	$("#after").on("click",function(){
    	$("#demo").css("clear","both");
    });
</script>
<br>上面的例子中由于div1的浮动，div2的一部分被浮动的元素覆盖，容器的高度比它里面浮动的元素高度还要低。</p>
<span id="more"></span>
<h3 id="二-使用clear属性清除浮动"><a href="#二-使用clear属性清除浮动" class="headerlink" title="二.使用clear属性清除浮动"></a>二.使用clear属性清除浮动</h3><p>清除浮动的方式有千千万，但其思路一共就两种，其一是使用CSS的clear属性，clear属性规定元素的哪一侧不允许出现浮动元素，在CSS1,CSS2中浏览器自动为带有clear属性的元素添加额外的margin-top，从而保证该元素不会与浮动的元素重叠在一起，CSS2.1规范改成在clear元素的外边距之上添加清除空间，因此不会改变元素的外边距，但无论哪种实现，其目的都是为浮动的元素留出足够的垂直空间。在应用clear属性之前普通流中的元素表现得就像浮动元素不存在一样，而应用该属性后则能“感受”到浮动元素的高度，最后完成清除浮动。<br>现在可以回头再看看上面的例子，查看网页源代码会发现js动态的为div2指定了clear属性，div2感知到了div1的存在从而解决了元素重叠的问题，由于div2处于普通流，容器必须扩展自身的高度才能够包裹住div2，这又解决了第二个问题。现在的问题是这种方式需要div2的辅助，才能清除div1的浮动，而实际应用中有可能浮动的元素本身就是容器的最后一个元素，换句话说实际应用中可能没有div2的存在，此时如何处理容器高度坍塌的问题？<br>简单粗暴的方式是人为的在浮动的元素后添加一个没有意义的空元素，并为该元素指定clear属性，这就是清除浮动代码的第一个版本，代码如下：<br><figure class="highlight html"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="tag">&lt;<span class="name">div</span> <span class="attr">class</span>=<span class="string">&quot;container&quot;</span>&gt;</span></span><br><span class="line">    <span class="tag">&lt;<span class="name">div</span> <span class="attr">style</span>=<span class="string">&quot;float:left&quot;</span>&gt;</span>我浮动了<span class="tag">&lt;/<span class="name">div</span>&gt;</span></span><br><span class="line">    <span class="tag">&lt;<span class="name">div</span> <span class="attr">style</span>=<span class="string">&quot;clear:both&quot;</span>&gt;</span><span class="comment">&lt;!--我用来清除上面的浮动--&gt;</span><span class="tag">&lt;/<span class="name">div</span>&gt;</span></span><br><span class="line"><span class="tag">&lt;/<span class="name">div</span>&gt;</span></span><br></pre></td></tr></table></figure><br>上面的代码有两个问题，一是添加无意义的标签给开发与维护带来麻烦，二是违反了web开发的原则，即html用来决定网页的内容，css用来决定呈现的样式。上面的代码添加了一个html元素却仅仅用来改变呈现的样式。于是大家顺其自然的想到使用css为<strong>容器</strong>添加一个:after伪元素，代码变成了下面这样：<br><figure class="highlight css"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="selector-class">.clearfix</span><span class="selector-pseudo">:after</span>&#123;</span><br><span class="line">  <span class="attribute">content</span>:<span class="string">&quot;&quot;</span>;</span><br><span class="line">  <span class="attribute">display</span>:block;</span><br><span class="line">  <span class="attribute">clear</span>:both;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br><figure class="highlight html"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="tag">&lt;<span class="name">div</span> <span class="attr">class</span>=<span class="string">&quot;container clearfix&quot;</span>&gt;</span></span><br><span class="line">  <span class="tag">&lt;<span class="name">div</span> <span class="attr">style</span>=<span class="string">&quot;float:left&quot;</span>&gt;</span>我浮动了<span class="tag">&lt;/<span class="name">div</span>&gt;</span></span><br><span class="line">  <span class="comment">&lt;!--这里会有css创建的伪元素，用来清除上面的浮动--&gt;</span></span><br><span class="line"><span class="tag">&lt;<span class="name">div</span>&gt;</span></span><br></pre></td></tr></table></figure><br>至此，这个浮动清除类(clearfix)已经足以应付大多数情况了。但仍有修改的余地。</p>
<h3 id="三-使用块级格式化上下文-Block-formatting-contexts-清除浮动"><a href="#三-使用块级格式化上下文-Block-formatting-contexts-清除浮动" class="headerlink" title="三.使用块级格式化上下文(Block formatting contexts)清除浮动"></a>三.使用块级格式化上下文(Block formatting contexts)清除浮动</h3><h4 id="1-BFC有什么用"><a href="#1-BFC有什么用" class="headerlink" title="1.BFC有什么用"></a>1.BFC有什么用</h4><p>CSS2.1开始引入BFC的概念，它是KFC的兄弟品牌，额，不好意思脑袋岔路了，通俗的讲BFC指的是页面中的一块渲染区域，它有自己独立的渲染规则，区域中的元素布局不会受到外部影响，也不会影响外部元素。英语基础好的同学还可以点开链接阅读w3c官方对<a href="https://www.w3.org/TR/CSS21/visuren.html#block-formatting">Block formatting contexts</a>的定义，然后你会发现它几乎没什么实质性的内容，这里强烈推荐YUI团队博客的<a href="http://yuiblog.com/blog/2010/05/19/css-101-block-formatting-contexts/">Block Formatting Contexts</a>这篇文章，它清晰的解答了BFC在实际开发中起什么作用：</p>
<ul>
<li><p>BFC决定了元素间是否会发生垂直外边距叠加<br>  只有处于同一上下文中的块框才有可能会发生垂直外边距叠加，参考下面的例子：</p>
  
    <div><p style="background:skyblue;margin:10px 0;">p1 我有10px的margin</p></div>
    <div><p style="background:skyblue;margin:10px 0;">p2 我有10px的margin</p></div>
    <div style="overflow:hidden;"><p style="background:skyblue;margin:10px 0;">p3 我也有10px的margin，但我与楼上二位不在同一BFC</p></div>
    
<p>  上例中p1,p2都有20px的margin，但由于外边距叠加，它们的间距仍然是10px，而p3的容器指定了overflow属性(后面解释)，创建了新的BFC，导致了p3的垂直外边距没有发生叠加，p2与p3的间距为20px。</p>
</li>
<li><p>BFC不会与浮动元素发生重叠<br>  更确切的说法是BFC的边框不会与其它浮动元素的外边距发生重叠，这意味着浏览器可能会给BFC创建隐式的外边距来实现该效果，参考下面的例子：</p>
  
    <div>
    	<div style="background:skyBlue;float:left;width:180px;height:100px">我向左浮动(width:180px)</div>
    	<div style="background:pink;float:right;width:180px;height:100px">我向右浮动(width:180px)</div>
        <div style="background:wheat;border:5px solid #a57958;height:100px">div1
        虽然文字没有被覆盖，但我与浮动元素重叠了</div><br/>
    	<div style="background:skyBlue;float:left;width:180px;height:100px">我向左浮动(width:180px)</div>
    	<div style="background:pink;float:right;width:180px;height:100px">我向右浮动(width:180px)</div>
        <div style="background:wheat;overflow:hidden;border:5px solid #a57958;height:100px">div2
        我创建了BFC，我与浮动元素没有发生重叠</div>
    </div><br/>
    
<p>  上面的例子中，浏览器为div2添加隐式的外边距来保证它的border-box不会与其它浮动的元素发生重叠，换句话说div2出场就自带了<code>margin:0 180px;</code>的特效，此时你为div2手动指定任何小于180px的水平外边距将得不到任何效果。此时如果你想在div2与浮动的元素之间添加10px的空隙，你需要为div2指定190px的水平外边距。<br>  最后，有人可能会想，这里的div2通过压缩自身的宽度从而保证了不与浮动元素发生重叠，假设我为div2指定一个较大的固定宽度，它将如何应对？答案是它会被挤到下面一行，依然不会发生重叠。</p>
</li>
<li><p>BFC会包含它里面浮动的元素<br>  对此w3c有明确的定义，参见<a href="https://www.w3.org/TR/CSS2/visudet.html#root-height">&#39;Auto&#39; heights for block formatting context roots</a>，其中核心的一句话是“如果BFC根元素包含浮动的后代元素，并且该后代元素的bottom margin edge低于元素的bottom content edge，则元素自动增加高度直到能够包含这些边缘。”原谅我蹩脚的翻译，还是直接来看例子吧：</p>
  
    <div style="border:5px solid #a57958;margin-top:10px;">
        div1 我是容器
        <div style="color:black;background-color:wheat;padding:10px;width:30%;height:100px;float:left;margin:10px">我浮动了(margin:10px)</div>
    </div>
    <br clear="both"/>
    <div style="border:5px solid #a57958;margin-top:10px;overflow:hidden;">
        div2 我是BFC容器
        <div style="color:black;background-color:wheat;padding:10px;width:30%;height:100px;float:left;margin:10px">我浮动了(margin:10px)</div>
    </div><br/>
    
<p>  上面例子中div2创建了新的块级格式化上下文，因此得以将浮动的子元素(包括其margin)包含其中。回顾一下文章的主题，这不正是我们所要的清除浮动效果吗？那么剩下的问题就是，我们应该如何让所有会包含浮动子元素的容器元素创建新的BFC。</p>
</li>
</ul>
<h4 id="2-如何创建BFC"><a href="#2-如何创建BFC" class="headerlink" title="2.如何创建BFC"></a>2.如何创建BFC</h4><p>下列任一情况将创建一个块格式化上下文（资料来源于MDN）：</p>
<ul>
<li>根元素或其它包含它的元素</li>
<li>浮动 (float:left|right)</li>
<li>绝对定位元素 (position:absolute|fixed)</li>
<li>display属性(display:inline-block|table-cell|table-caption|flex|inline-flex)</li>
<li>overflow除visible外的任意值(overflow:hidden|auto|scroll)</li>
</ul>
<h4 id="3-如何通过BFC清除浮动"><a href="#3-如何通过BFC清除浮动" class="headerlink" title="3.如何通过BFC清除浮动"></a>3.如何通过BFC清除浮动</h4><p>正如上一节看到的，创建BFC的方法有多少种，清除浮动的方法就有多少种。一开始有人尝试通过为容器指定float属性创建BFC，我只是想想都觉得不靠谱，清除浮动的方法居然是将所有元素都浮动起来。一种稍微靠谱的方法是将容器的overflow属性设为hidden或者auto，比起修改position与display，它的副作用(截断内容或滚动条)已经算是可以接受的了，但我们最开始的clear伪元素可是没有任何副作用的。好吧，这确实有点让人沮丧，尽管花了这么长的篇幅来说明BFC，但它还不如直接使用clear伪元素。幸运的是在clearfix最佳实践中，还会用到它的概念。</p>
<h3 id="四-clearfix最佳实践与原理"><a href="#四-clearfix最佳实践与原理" class="headerlink" title="四.clearfix最佳实践与原理"></a>四.clearfix最佳实践与原理</h3><p>回顾第二段，现在我们的clearfix版本是这样的：<br><figure class="highlight css"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="selector-class">.clearfix</span><span class="selector-pseudo">:after</span>&#123;</span><br><span class="line">  <span class="attribute">content</span>:<span class="string">&quot;&quot;</span>;</span><br><span class="line">  <span class="attribute">display</span>:block;</span><br><span class="line">  <span class="attribute">clear</span>:both;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br>下面则是目前CSS清除浮动的最佳实践，来自Nicolas Gallagher，详细内容参见<a href="http://nicolasgallagher.com/micro-clearfix-hack/">A new micro clearfix hack</a><br><figure class="highlight css"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="selector-class">.clearfix</span><span class="selector-pseudo">:before</span>,<span class="selector-class">.clearfix</span><span class="selector-pseudo">:after</span>&#123;</span><br><span class="line">  <span class="attribute">content</span>:<span class="string">&quot; &quot;</span>; <span class="comment">/*注意里面的空格*/</span></span><br><span class="line">  <span class="attribute">display</span>:table;</span><br><span class="line">&#125;</span><br><span class="line"><span class="selector-class">.clearfix</span><span class="selector-pseudo">:after</span>&#123;</span><br><span class="line">  <span class="attribute">clear</span>:both;</span><br><span class="line">&#125;</span><br><span class="line"><span class="selector-class">.clearfix</span>&#123;</span><br><span class="line">  zoom:<span class="number">1</span>; <span class="comment">/* For IE 6/7 (trigger hasLayout) */</span></span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br>对比上面两个版本的不同，很容易提出下面几个问题。</p>
<h4 id="1-content里面为什么需要一个空格？"><a href="#1-content里面为什么需要一个空格？" class="headerlink" title="1.content里面为什么需要一个空格？"></a>1.content里面为什么需要一个空格？</h4><p>根据原文的描述，在content添加空格是为了处理一个Opera浏览器的显示bug，由于这个问题我在自己的Opera上没有重现，因此也无法给出更详细的说明。</p>
<h4 id="2-zoom-1有什么用？"><a href="#2-zoom-1有什么用？" class="headerlink" title="2.zoom:1有什么用？"></a>2.zoom:1有什么用？</h4><p>由于伪元素与BFC是由CSS2.1规范提出，因此上面讨论的两种方式IE6与IE7均不支持，但它们有一个与BFC类似的概念叫hasLayout，限于篇幅，这里不再详细讨论，简单的说它同样具备BFC的作用即：</p>
<ul>
<li>决定外边距是否折叠</li>
<li>不与浮动元素发生重叠</li>
<li>包含它里面的浮动元素</li>
</ul>
<p>它的触发条件是(满足任一即可)：</p>
<ul>
<li>float:left|right</li>
<li>position:absolute|fixed</li>
<li>display:inline-block</li>
<li>writing-mode:tb-rl</li>
<li>width:(除auto外任意值)</li>
<li>height:(除auto外任意值)</li>
<li>zoom:(除normal外任意值)</li>
</ul>
<p>IE7还包括：</p>
<ul>
<li>min-width|max-width:(任意值)</li>
<li>min-height|max-height:(任意值)</li>
<li>overflow:(除visible外任意值)</li>
<li>overflow-x|overflow-y:(除visible外任意值)</li>
</ul>
<p>因此这里通过<code>zoom:1</code>触发IE6,IE7的hasLayout，其原理与创建BFC是一样的。</p>
<h4 id="3-before伪元素有什么用？display属性为什么要设成table？"><a href="#3-before伪元素有什么用？display属性为什么要设成table？" class="headerlink" title="3.before伪元素有什么用？display属性为什么要设成table？"></a>3.before伪元素有什么用？display属性为什么要设成table？</h4><p>把它们放在一起是因为它们的目的都是为了阻止垂直外边距叠加。举个例子：<br>
<div style="background-color:skyblue;margin:20px 0;">
	<div style="display:block;"></div>
	<div style="background-color:wheat;margin:20px;height:50px;">div1
    我的垂直外边距与容器叠加(margin:20px)</div>
	<div style="display:block"></div>
</div>
<div style="background-color:skyblue;margin:20px 0;">
	<div style="display:table"></div>
	<div style="background-color:wheat;margin:20px;height:50px">div2
    我的垂直外边距不与容器叠加(margin:20px)</div>
	<div style="display:table"></div>
</div>
<br>它的原理用到了之前提到的BFC概念，display:table创建了匿名的display:table-cell框，而前面已经提到，display:table-cell将创建新的BFC，从而阻止了垂直外边距叠加。然而也许你已经意识到了，它跟清除浮动有一毛钱关系吗？是否需要垂直外边距叠加本应该由各自网站的开发人员来决定，换句话说为什么要在清除浮动的代码中强行加入设定垂直外边距是否叠加的代码？这就像通过设定overflow:hidden来清除浮动一样产生了副作用。<br>其实它是为了浏览器显示一致性而添加的代码，前面已经讨论过，假设我们通过创建BFC来清除浮动，那么它的垂直外边距不会叠加。同理，通过触发hasLayout也不会叠加。因此如果没有<code>clearfix:before</code>与<code>display:table</code>，那么元素的垂直外边距在现代浏览器下会叠加，而在IE6与IE7中不会叠加，这就破坏了样式的一致性。换句话说，如果不需要支持IE6与IE7，那么使用如下代码就足够了：<br><figure class="highlight css"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="selector-class">.clearfix</span><span class="selector-pseudo">:after</span>&#123;</span><br><span class="line">  <span class="attribute">content</span>:<span class="string">&quot; &quot;</span>; <span class="comment">/*注意里面的空格*/</span></span><br><span class="line">  <span class="attribute">display</span>:block;</span><br><span class="line">  <span class="attribute">clear</span>:both;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure></p>
<p>完！</p>
]]></content>
      
        <categories>
            
            <category> Frontend </category>
            
        </categories>
        
        
        <tags>
            
            <tag> CSS </tag>
            
        </tags>
        
    </entry>
    
    <entry>
      <title><![CDATA[为什么bootstrap采用border-box盒模型]]></title>
      <url>https://blog.staynoob.cn/post/2016/05/%25E4%25B8%25BA%25E4%25BB%2580%25E4%25B9%2588bootstrap%25E9%2587%2587%25E7%2594%25A8border-box%25E7%259B%2592%25E6%25A8%25A1%25E5%259E%258B/</url>
      <content type="html"><![CDATA[<blockquote>
<p>作为前端萌新，想要制作自己的博客样式，自然首先想到bootstrap框架。在开发过程中我发现，bootstrap将全局范围的box-sizing属性重置成了border-box，而不是w3c默认的content-box。google中文结果没有查到bootstrap这么做的原因，甚至有人说这是bootstrap的一个坑(囧)，于是这里我把我查到的一些资料记录下来。</p>
</blockquote>
<p>CSS盒模型指的是由content,padding,border,margin组成的框结构，相关内容每一本CSS书籍都有详细介绍，这里就不多说了，这篇文章主要讲的是当我们为元素指定width,height属性时，将如何影响到元素的显示效果。文章主要参考的是<a href="https://css-tricks.com/box-sizing/">box-sizing</a>这篇文章，建议有英语基础的同学直接阅读该篇文章。</p>
<h3 id="一-CSS盒模型历史"><a href="#一-CSS盒模型历史" class="headerlink" title="一.CSS盒模型历史"></a>一.CSS盒模型历史</h3><p>在CSS起源时期，w3c规范CSS盒模型默认按如下方式计算：</p>
<ul>
<li>width + padding + border = 元素实际可见的宽度</li>
<li>height + padding + border = 元素实际可见的高度</li>
</ul>
<p>这样的设计其实有点反人类，因为当你经过周密计算，将元素的宽度，高度设定好，并将其定位后。假设领导不满意，打算让你将border或padding加粗一个像素，有可能你就需要重新调整布局。IE浏览器很早就看到了这个隐患，并勇敢的站出来挑战规范！在IE5中实现了一个不一样的盒模型，它按如下方式来计算：</p>
<ul>
<li>width = 元素实际可见的宽度</li>
<li>height = 元素实际可见的高度</li>
</ul>
<p>这里面的width不再是content的宽度，而是包含了content,padding,border的总和，当你为元素添加的边框与内补越多，给内容留下的空间也就越少，这其实非常合理，并且更有利于开发人员的布局。然而它却不符合规范。于是微软在IE6版本就及时向规范妥协了，在此后的版本(IE6 - IE8)中，仅仅在“混杂模式(quirks mode)”下继续使用这个模型。顺便解释一下混杂模式，浏览器开发通常既要考虑到最新的web规范，又要考虑向后兼容，因此IE的开发者创建了两种呈现模式，即标准模式与混杂模式，前者主要用于兼容最新规范，后者主要用于兼容老的IE版本，在HTML代码中不使用或使用错误的DOCTYPE将进入混杂模式。</p>
<p>尽管规范中的盒模型有这样的缺陷，但是当人们理解并习惯了之后，发现用该模型进行开发也不算太难，只要在给元素设定尺寸的时候事先计算一下大概要给padding与border预留多少空间就可以了。然而当web开发进入响应式设计(Responsive Web Design)的时代后，相同的方式就不灵了，因为预留的固定像素无法自适应屏幕大小。</p>
<span id="more"></span>
<h3 id="二-CSS3的box-sizing属性"><a href="#二-CSS3的box-sizing属性" class="headerlink" title="二.CSS3的box-sizing属性"></a>二.CSS3的box-sizing属性</h3><p>为了适应移动互联网的浪潮，CSS3在提升响应式开发体验上下了很多功夫，比如说添加了media query媒介查询模块(CSS2.1中的@media属性只能用来区分媒体的类型，无法查询具体屏幕尺寸)，另一项重要的改进就是引入box-sizing属性来决定盒模型的渲染方式，它有三个可选值，分别是：content-box(默认值),padding-box,border-box。其中padding-box几乎没人用，content-box依然按照规范工作，而border-box则采取与老版本IE混杂模式中对盒模型相同的解释方式，这也是最流行的选项。看到这里，你也许仍然不明白border-box到底好在哪里，下面是一个Demo:</p>

<div style="border:5px solid #a57958;padding:10px">
	<p>按下按钮查看box-sizing属性对布局的影响</p>
    <button class="demo-btn btn btn-info">content-box</button>
    <button class="demo-btn btn btn-info">border-box</button>
    <div id="demo-container" class="clearfix" style="margin-top:10px">
    	<div style="margin:0 auto 5px;padding:10%;width:90%;border:5px solid #a57958">width=90%
        padding=10%
        border=5px</div>
    	<div style="float:left;width:50%;border:5px solid #a57958;height:100px;">width=50%
        border=5px</div>
    	<div style="float:left;width:50%;border:5px solid #a57958;height:100px;">width=50%
        border=5px</div>
    </div>
</div>
<script>
	$(".demo-btn").on("click",function(){
    	$("*").css("box-sizing",$(this).text());
    });
</script>

<h3 id="三-重置方法最佳实践"><a href="#三-重置方法最佳实践" class="headerlink" title="三.重置方法最佳实践"></a>三.重置方法最佳实践</h3><p>现在，我们已经知道了使用border-box盒模型进行布局更加简单，而除非你的代码是面向IE5用户开发的，否则都需要使用重置代码来修改默认的盒模型，最早人们使用通配符选择器(universal type selector)来重置所有元素的box-sizing属性。代码如下：<br><figure class="highlight css"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">* &#123;</span><br><span class="line">  <span class="attribute">box-sizing</span>:border-box;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br>大部分情况下上述代码就够了，但它还遗漏了伪元素，改进后的代码如下：<br><figure class="highlight css"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">*,*<span class="selector-pseudo">:before</span>,*<span class="selector-pseudo">:after</span>&#123;</span><br><span class="line">  <span class="attribute">box-sizing</span>:border-box;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br>以上也是bootstrap(v3.3.5)中使用的重置代码，它已经完全够用了，不过假如你的需求非常变态，想要在网页中混合使用content-box,padding-box,border-box,可以采用如下的重置代码：<br><figure class="highlight css"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="selector-tag">html</span>&#123;</span><br><span class="line">  <span class="attribute">box-sizing</span>:border-box;</span><br><span class="line">&#125;</span><br><span class="line">*,*<span class="selector-pseudo">:before</span>,*<span class="selector-pseudo">:after</span>&#123;</span><br><span class="line">  <span class="attribute">box-sizing</span>:inherit;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br>上面的代码可以让你的选择更加有弹性，可以为不同的布局容器指定各自的盒模型，而不用担心每个容器的子元素被全局样式重置。</p>
<h3 id="四-浏览器支持"><a href="#四-浏览器支持" class="headerlink" title="四.浏览器支持"></a>四.浏览器支持</h3><p>所有现代浏览器已经全部支持无前缀的box-sizing属性，如果你想要支持一些老到几乎没人用的浏览器版本(safari &lt; 5.1,chrome &lt; 10,firefox &lt; 29)，可以像bootstrap一样为该属性加上浏览器前缀：<br><figure class="highlight css"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">*,*<span class="selector-pseudo">:before</span>,*<span class="selector-pseudo">:after</span>&#123;</span><br><span class="line">  -webkit-<span class="attribute">box-sizing</span>:border-box;</span><br><span class="line">  -moz-<span class="attribute">box-sizing</span>:border-box;</span><br><span class="line">  <span class="attribute">box-sizing</span>:border-box;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br>不过padding-box因为没什么人用，所以一些浏览器可能会不支持。IE7及其以下版本不支持box-sizing属性。</p>
]]></content>
      
        <categories>
            
            <category> Frontend </category>
            
        </categories>
        
        
        <tags>
            
            <tag> CSS </tag>
            
        </tags>
        
    </entry>
    
    <entry>
      <title><![CDATA[聊聊4种主流排序算法(番外篇):快速排序的优化历程]]></title>
      <url>https://blog.staynoob.cn/post/2016/05/%25E8%2581%258A%25E8%2581%258A4%25E7%25A7%258D%25E4%25B8%25BB%25E6%25B5%2581%25E6%258E%2592%25E5%25BA%258F%25E7%25AE%2597%25E6%25B3%2595-%25E7%2595%25AA%25E5%25A4%2596%25E7%25AF%2587-%25E5%25BF%25AB%25E9%2580%259F%25E6%258E%2592%25E5%25BA%258F%25E7%259A%2584%25E4%25BC%2598%25E5%258C%2596%25E5%258E%2586%25E7%25A8%258B/</url>
      <content type="html"><![CDATA[<blockquote>
<p>这篇文章主要介绍的是快速排序的一些公认有效的优化方法，它们中间有很多都来自于实践经验(empirical)。</p>
</blockquote>
<p>在进行优化之前，我觉得必要先找到优化的起点，这里直接使用上篇文章的实现版本作为最原始的版本，如果这段代码你无法理解，请先阅读上一篇文章。看过上篇文章则可以直接跳过这部分。</p>
<span id="more"></span>
<figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">void</span> <span class="title function_">quickSort</span><span class="params">(<span class="type">int</span>[] arr)</span>&#123;</span><br><span class="line">    quickSort(arr,<span class="number">0</span>,arr.length - <span class="number">1</span>);</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="keyword">private</span> <span class="keyword">static</span> <span class="keyword">void</span> <span class="title function_">quickSort</span><span class="params">(<span class="type">int</span>[] arr,<span class="type">int</span> lo,<span class="type">int</span> hi)</span>&#123;</span><br><span class="line">    <span class="keyword">if</span>(lo&gt;=hi) <span class="keyword">return</span>;</span><br><span class="line">    <span class="type">int</span> pivot=partition(arr,lo,hi);</span><br><span class="line">    quickSort(arr,lo,pivot - <span class="number">1</span>);</span><br><span class="line">    quickSort(arr,pivot + <span class="number">1</span>,hi);</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="comment">/**</span></span><br><span class="line"><span class="comment">*直接选取第一个元素lo作为pivot，使用两个指针(i,j)扫描数组，i,j朝数组中间移动</span></span><br><span class="line"><span class="comment">*遇到arr[i]&gt;=pivot则i停止，遇到arr[j]&lt;=pivot则j停止，交换i,j对应的元素，重复该过程直到i,j相遇</span></span><br><span class="line"><span class="comment">*当整个过程停下来时将pivot与a[j]交换位置，从而完成分区</span></span><br><span class="line"><span class="comment">*</span></span><br><span class="line"><span class="comment">*       left part              center part           right part</span></span><br><span class="line"><span class="comment">* +--------------------------------------------------------------+</span></span><br><span class="line"><span class="comment">* lo |   &lt;= pivot   |              ?              |   &gt;= pivot   |</span></span><br><span class="line"><span class="comment">* +--------------------------------------------------------------+</span></span><br><span class="line"><span class="comment">* ^                 ^                             ^</span></span><br><span class="line"><span class="comment">* |                 |                             |</span></span><br><span class="line"><span class="comment">* pivot             i-&gt;                         &lt;-j</span></span><br><span class="line"><span class="comment">*</span></span><br><span class="line"><span class="comment">*/</span></span><br><span class="line"><span class="keyword">private</span> <span class="keyword">static</span> <span class="type">int</span> <span class="title function_">partition</span><span class="params">(<span class="type">int</span>[] arr,<span class="type">int</span> lo,<span class="type">int</span> hi)</span>&#123;</span><br><span class="line">    <span class="type">int</span> i=lo+<span class="number">1</span>,j=hi;</span><br><span class="line"></span><br><span class="line">    <span class="keyword">while</span>(<span class="literal">true</span>)&#123;</span><br><span class="line">        <span class="keyword">while</span>(arr[++i]&lt;arr[lo] &amp;&amp; i &lt; hi);</span><br><span class="line">        <span class="keyword">while</span>(arr[--j]&gt;arr[lo]); <span class="comment">//这里不需要判断j&gt;lo，因为当j=lo时，arr[j]&gt;arr[lo]不可能成立</span></span><br><span class="line">        <span class="keyword">if</span>(i&gt;=j) <span class="keyword">break</span>; 		 <span class="comment">//i,j相遇时停止</span></span><br><span class="line">        exchange(arr,i,j);</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    exchange(arr,lo,j);</span><br><span class="line">    <span class="keyword">return</span> j;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure>
<h3 id="一-在内循环中加入插入排序-cut-off-to-insertion-sort"><a href="#一-在内循环中加入插入排序-cut-off-to-insertion-sort" class="headerlink" title="一.在内循环中加入插入排序(cut off to insertion sort)"></a>一.在内循环中加入插入排序(cut off to insertion sort)</h3><p>这其实没什么好说的，<a href="http://blog.staynoob.cn/post/algorithm/insertion-shell-merge-quick-sort-algorithm-1/#三-插入-Insertion-排序">插入排序</a>章节中已经给出了明确的解释。在快速排序，归并排序等基于递归的排序算法中都可以加入插入排序来处理小规模数组，而对“小”的定义则需要考虑具体的系统环境，编程语言等因素，算法书上给出了5-20的建议。下面是改进后的代码片段：<br><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">private</span> <span class="keyword">static</span> <span class="keyword">final</span> <span class="type">int</span> CUTOFF=<span class="number">8</span>;</span><br><span class="line"><span class="keyword">private</span> <span class="keyword">static</span> <span class="keyword">void</span> <span class="title function_">quickSort</span><span class="params">(<span class="type">int</span>[] arr,<span class="type">int</span> lo,<span class="type">int</span> hi)</span>&#123;</span><br><span class="line">    <span class="keyword">if</span>(hi - lo + <span class="number">1</span> &lt;= CUTTOFF ) &#123;</span><br><span class="line">    	insertionSort(arr,lo,hi);</span><br><span class="line">        <span class="keyword">return</span>;</span><br><span class="line">    &#125;</span><br><span class="line">    <span class="type">int</span> pivot=partition(arr,lo,hi);</span><br><span class="line">    quickSort(arr,lo,pivot - <span class="number">1</span>);</span><br><span class="line">    quickSort(arr,pivot + <span class="number">1</span>,hi);</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure></p>
<h3 id="二-三数中值切分-median-of-three-partitioning"><a href="#二-三数中值切分-median-of-three-partitioning" class="headerlink" title="二.三数中值切分(median of three partitioning)"></a>二.三数中值切分(median of three partitioning)</h3><p>在原始版本中我们直接使用数组的第一个元素作为pivot(枢纽元)，成功的在数组已经预排序的情况下将复杂度提升到了平方级，这意味着消耗大量时间却什么也没干。针对这个问题大家尝试了一些解决方案：</p>
<ul>
<li>数组中间随机挑一个元素作为pivot，不得不说这在实践中比直接选取第一个元素要好，但如果输入数组是随机的，数组已经预排序的概率与其它分布情况一致，则该方法起不到任何效果，反而还要消耗生成随机数的时间。</li>
<li>寻找数组的中位素(median)，每次都使用中位数作为pivot是快速排序的最好情况，但实践中发现在内循环中寻找中位数的开销远远大于其省下来的时间。</li>
<li>既然寻找整个数组的中位数不靠谱，那要不试试在数组中随机取三个数，再取它们的中位数作为pivot?oh!It works!I&#39;m such a genius!...实践证明这种方法非常不错，然而在查到的资料中都没有明确的说明为什么，它们给出的解释大多都是“实验证明(empirical)”。</li>
<li>没有三取样解决不了的问题，如果有，那就取九个，在我的算法书上称其为&quot;Tukey&#39;s ninther&quot;，又称&quot;median of medians&quot;。即先选出三组元素，每组三个，分别取三组元素的中位数，再从这三个中位数中取其中位数作为pivot，这在输入数组比较大时，进一步降低了切分不均匀的概率。</li>
</ul>
<p>也许关于这个问题我的描述带有过多的调侃成分，这完全出于我没有办法量化三数中值切分所带来的好处，而我又不想给出一个模棱两可的回答，《数据结构与算法分析》这本书说采取三数中值切分减少了快速排序大约5%的时间，请你也尝试说服自己接受这个答案。无论如何它总归可以避免快速排序沦为平方级算法，下面是改进后的代码片段：<br><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">private</span> <span class="keyword">static</span> <span class="keyword">void</span> <span class="title function_">quickSort</span><span class="params">(<span class="type">int</span>[] arr,<span class="type">int</span> lo,<span class="type">int</span> hi)</span>&#123;</span><br><span class="line">    <span class="type">int</span> length=hi-lo+<span class="number">1</span>;</span><br><span class="line"></span><br><span class="line">    <span class="keyword">if</span>(length &lt;= CUTTOFF ) &#123;</span><br><span class="line">    	<span class="comment">//use insertionSort</span></span><br><span class="line">    	insertionSort(arr,lo,hi);</span><br><span class="line">        <span class="keyword">return</span>;</span><br><span class="line">    &#125;<span class="keyword">else</span> <span class="keyword">if</span>(length &lt;= <span class="number">40</span>)&#123;</span><br><span class="line">    	<span class="comment">//use median of three</span></span><br><span class="line">    	<span class="type">int</span> <span class="variable">m</span> <span class="operator">=</span> median3(arr, lo, lo + N&gt;&gt;<span class="number">1</span>, hi);</span><br><span class="line">        exchange(arr, lo, m);</span><br><span class="line">    &#125;<span class="keyword">else</span> &#123;</span><br><span class="line">    	<span class="comment">//use median of medians</span></span><br><span class="line">    	<span class="type">int</span> <span class="variable">eps</span> <span class="operator">=</span> N/<span class="number">8</span>;</span><br><span class="line">        <span class="type">int</span> <span class="variable">mid</span> <span class="operator">=</span> lo + N&gt;&gt;<span class="number">1</span>;</span><br><span class="line">        <span class="type">int</span> <span class="variable">m1</span> <span class="operator">=</span> median3(arr, lo, lo + eps, lo + eps + eps);</span><br><span class="line">        <span class="type">int</span> <span class="variable">m2</span> <span class="operator">=</span> median3(arr, mid - eps, mid, mid + eps);</span><br><span class="line">        <span class="type">int</span> <span class="variable">m3</span> <span class="operator">=</span> median3(arr, hi - eps - eps, hi - eps, hi);</span><br><span class="line">        <span class="type">int</span> <span class="variable">ninther</span> <span class="operator">=</span> median3(arr, m1, m2, m3);</span><br><span class="line">        exchange(arr,lo,ninther);</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    <span class="type">int</span> pivot=partition(arr,lo,hi);</span><br><span class="line">    quickSort(arr,lo,pivot - <span class="number">1</span>);</span><br><span class="line">    quickSort(arr,pivot + <span class="number">1</span>,hi);</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="keyword">private</span> <span class="keyword">static</span> <span class="type">int</span> <span class="title function_">median3</span><span class="params">(<span class="type">int</span>[] arr,i,j,k)</span>&#123;</span><br><span class="line">    <span class="keyword">return</span>  (arr[i] &lt; arr[j] ?</span><br><span class="line">            (arr[j] &lt; arr[k] ? j : arr[i] &lt; arr[k] ? k : i):</span><br><span class="line">            (arr[j] &lt; arr[k] ? k : arr[i] &lt; arr[k] ? i : k));</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure></p>
<h3 id="三-三向切分快速排序-three-way-quick-sort"><a href="#三-三向切分快速排序-three-way-quick-sort" class="headerlink" title="三.三向切分快速排序(three way quick sort)"></a>三.三向切分快速排序(three way quick sort)</h3><p>实际应用中，数组常常包含一些重复元素，到目前为止我们的排序算法在应对这种数组时显然是不够优化的，即便是处理一个全是重复元素的数组，我们的算法还是把它视为普通输入一样切分，递归。而三向切分快速排序正是用于处理这个问题，它将数组切分成三部分，分别是小于pivot，等于pivot，大于pivot。在之前的实现中，每次递归至少能保证将pivot元素放在正确的位置，而应用此项改进后，每次都能将所有等于pivot的元素放到正确位置。在研究它的思路之前，可以先来看看由Dijkstra提出的Dutch National Flag Problem(荷兰国旗问题)，下面是简单描述版本。<br>假设一个数组分别由A,B,C三个元素组成，三个元素随机排列，且每个元素的个数不确定，应该如何重新排列该数组使其恢复有序?<br>Example:<br>Input=[A,B,B,C,A,C,B];<br>Output=[A,A,B,B,B,C,C];<br>这样看起来问题非常简单，使用一个指针按顺序扫描数组，遇到A则将其换到左边，遇到C则将其换到右边。这跟最原始的三向切分思路一模一样，下面是其实现代码，你也可以在JDK1.7以上版本双枢纽快速排序(Dual-Pivot-Quick-Sort)源码中看到它(不过只有在特定情形下才会调用它)：<br><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">/*</span></span><br><span class="line"><span class="comment"> *   left part    center part              right part</span></span><br><span class="line"><span class="comment"> * +--------------------------------------------------+</span></span><br><span class="line"><span class="comment"> * lo|  &lt; pivot  |   == pivot   |     ?    |  &gt; pivot |</span></span><br><span class="line"><span class="comment"> * +--------------------------------------------------+</span></span><br><span class="line"><span class="comment"> *               ^              ^          ^</span></span><br><span class="line"><span class="comment"> *               |              |          |</span></span><br><span class="line"><span class="comment"> *              less            k        great</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"><span class="type">int</span> pivot=lo,less=lo,great=hi;</span><br><span class="line"></span><br><span class="line"><span class="keyword">for</span> (<span class="type">int</span> <span class="variable">k</span> <span class="operator">=</span> lo+<span class="number">1</span>; k &lt;= great; k++) &#123;</span><br><span class="line">    <span class="keyword">if</span>(arr[k]==pivot) <span class="keyword">continue</span>;</span><br><span class="line"></span><br><span class="line">    <span class="keyword">if</span>(arr[k]&lt;pivot)&#123;</span><br><span class="line">        exchange(arr,k,less++);</span><br><span class="line">    &#125; <span class="keyword">else</span>&#123;</span><br><span class="line">        <span class="keyword">while</span> (arr[great]&gt;pivot) great--;</span><br><span class="line">        exchange(arr,k,great--);</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br><span class="line">sort(arr,lo,less - <span class="number">1</span>);</span><br><span class="line">sort(arr,great + <span class="number">1</span>,hi);</span><br></pre></td></tr></table></figure><br>如果你急不可耐的在我们的快排版本中加入以上代码，并使用随机生成的数组进行测试，你会发现它几乎没什么卵用，因为它的内循环中明显增加了元素间的交换次数，只要k指针遇到的元素不等于pivot，都会被交换。除非你的快速排序仅用于解决上面的荷兰旗帜问题，否则应用这段代码不会对效率有什么帮助，因为实际应用中大部分元素都不会等于pivot。<br>正因为上面的问题，该项策略一直不太流行，直到90年代，<a href="https://en.wikipedia.org/wiki/Jon_Bentley">Jon Bentley</a>与<a href="https://en.wikipedia.org/wiki/Douglas_McIlroy">Douglas McIlroy</a>使用了一个非常聪明的办法解决了这个问题，既然交换不相等的元素开销过大，为什么不改成交换相等的元素？下面是应用了Bentley &amp; McIlroy三向切分快速排序的版本，实践证明它确实有效，然而它的缺陷是代码变得复杂，你不必完全看懂，明白思路即可。<br><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">/*</span></span><br><span class="line"><span class="comment"> * Three Way Partitioning</span></span><br><span class="line"><span class="comment"> * By Bently and McIlroy</span></span><br><span class="line"><span class="comment"> *</span></span><br><span class="line"><span class="comment"> *   left part            center part          right part</span></span><br><span class="line"><span class="comment"> * +-------------------------------------------------------+</span></span><br><span class="line"><span class="comment"> * lo|  = pivot  |  &lt; pivot  |  ?  |  &gt; pivot  |  = pivot  |</span></span><br><span class="line"><span class="comment"> * +-------------------------------------------------------+</span></span><br><span class="line"><span class="comment"> *               ^           ^     ^           ^</span></span><br><span class="line"><span class="comment"> *               |           |     |           |</span></span><br><span class="line"><span class="comment"> *             less-&gt;        i-&gt;  &lt;-j        &lt;-great</span></span><br><span class="line"><span class="comment"> *</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"><span class="type">int</span> less=lo,great=hi+<span class="number">1</span>;</span><br><span class="line"></span><br><span class="line"><span class="keyword">while</span>(arr[++less]=arr[lo] &amp;&amp; less&lt;hi);</span><br><span class="line"><span class="keyword">while</span>(arr[--great]=arr[lo] &amp;&amp; great&gt;less);</span><br><span class="line"></span><br><span class="line"><span class="type">int</span> i=less-<span class="number">1</span>,j=great+<span class="number">1</span>;</span><br><span class="line"></span><br><span class="line"><span class="keyword">while</span> (<span class="literal">true</span>) &#123;</span><br><span class="line">    <span class="keyword">while</span>(arr[++i]&lt;arr[lo] &amp;&amp; i&lt;hi);</span><br><span class="line">    <span class="keyword">while</span>(arr[--j]&gt;arr[lo]);</span><br><span class="line"></span><br><span class="line">    <span class="keyword">if</span>(i==j) exchange(arr,i,less++);</span><br><span class="line">    <span class="keyword">if</span>(i&gt;=j) <span class="keyword">break</span>;</span><br><span class="line">    exchange(arr,i,j);</span><br><span class="line"></span><br><span class="line">    <span class="comment">//这里暂时将与pivot相等的元素换到数组的两端</span></span><br><span class="line">    <span class="keyword">if</span>(arr[i]=arr[lo]) exchange(arr,i,less++);</span><br><span class="line">    <span class="keyword">if</span>(arr[j]=arr[lo]) exchange(arr,j,great--);</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="comment">//这里将与pivot相等的元素换到正确位置</span></span><br><span class="line">i=j+<span class="number">1</span>;</span><br><span class="line"><span class="keyword">for</span> (<span class="type">int</span> <span class="variable">k</span> <span class="operator">=</span> lo; k &lt; less; k++) exchange(arr,k,j--);</span><br><span class="line"><span class="keyword">for</span> (<span class="type">int</span> <span class="variable">k</span> <span class="operator">=</span> hi; k &gt; great; k++) exchange(arr,k,i++);</span><br><span class="line"></span><br><span class="line">sort(arr,lo,j);</span><br><span class="line">sort(arr,i,hi);</span><br></pre></td></tr></table></figure><br>到这里快速排序的主要优化就已经结束了。</p>
<p>本文参考链接：</p>
<ul>
<li><a href="http://www.sorting-algorithms.com/static/QuicksortIsOptimal.pdf">QuicksortIsOptimal</a></li>
</ul>
]]></content>
      
        <categories>
            
            <category> Algorithm </category>
            
        </categories>
        
        
        <tags>
            
            <tag> Sort </tag>
            
        </tags>
        
    </entry>
    
    <entry>
      <title><![CDATA[聊聊4种主流排序算法(下)]]></title>
      <url>https://blog.staynoob.cn/post/2016/05/%25E8%2581%258A%25E8%2581%258A4%25E7%25A7%258D%25E4%25B8%25BB%25E6%25B5%2581%25E6%258E%2592%25E5%25BA%258F%25E7%25AE%2597%25E6%25B3%2595-%25E4%25B8%258B/</url>
      <content type="html"><![CDATA[<blockquote>
<p>上篇聊到了插入排序与希尔排序，它们分别擅长处理小型和中型输入，今天来聊两个适用于大型数组的排序算法，与上篇类似的是，这篇的两个算法设计思路也非常相似，它们都基于分治思想。</p>
</blockquote>
<h3 id="一-分治-Divide-and-Conquer"><a href="#一-分治-Divide-and-Conquer" class="headerlink" title="一.分治(Divide and Conquer)"></a>一.分治(Divide and Conquer)</h3><p>分治指的是将大问题拆分成相同类型的小问题，再联合小问题的解，从而最终解决大问题的一种算法设计思想。传统上认为分治算法必须将问题拆分成两个或多个子问题。因此二分查找不算分治算法。因为它只有单一子问题。下面先来一段“经典”代码：<br><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">public</span> <span class="keyword">static</span> <span class="type">int</span> <span class="title function_">fib</span><span class="params">(<span class="type">int</span> N)</span>&#123;</span><br><span class="line">    <span class="keyword">if</span>(N&lt;=<span class="number">1</span>) <span class="keyword">return</span> <span class="number">1</span>;</span><br><span class="line">    <span class="keyword">return</span> fib(N-<span class="number">1</span>)+fib(N-<span class="number">2</span>);</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br>上面这段代码依据给定N，计算斐波那契数列，之所以说它经典是因为这段代码作为反面教材出现在各大编程入门书籍中。它在字面上完全符合上面对分治的定义，但却效率低得惊人。作为子问题，fib(N-2)被重复计算了两次(fib(N-1)中也要计算fib(N-2))。因此它并没有将子问题完全“分开”。本文要讨论的两种排序算法跟上面的代码结构非常类似，但你必须要能够理解其中本质的不同。</p>
<span id="more"></span>
<h3 id="二-最快的排序算法"><a href="#二-最快的排序算法" class="headerlink" title="二.最快的排序算法"></a>二.最快的排序算法</h3><p>上一篇提到基于比较的最快的排序算法不可能优于O(NlogN)，你可能会好奇它是怎么来的，大牛们采用<a href="https://zh.wikipedia.org/wiki/%E4%BF%A1%E6%81%AF%E8%AE%BA">香农信息论(information theory)</a>证明，wiki上关于信息论的描述看完以后我反正是一个大写的懵B，这里提供一个更接地气的证明方法。假设有一个含有N个不重复元素的数组，那么它共有N!(阶乘)种不同状态，因为N个不同元素有N!种排列，每进行一次比较，比较结果有两种可能(大于或小于)，这将排除一些可能的状态，可以构造一棵二叉决策树来表示这个过程，下图表示对A,B,C三个元素排序的过程：</p>
<div align=center><br><img src="/img/content/decisiontree.jpg" alt="decisiontree.jpg"><br></div>

<p>这颗树的树叶表示所有可能的排序方式，而它的高度则是在最坏情况下需要的比较次数。不难推测出，对于给定的含有N个元素的数组，决策树必须要能够容纳下N!个叶节点，才能应付所有可能的情况，否则一定能构造一些输入，导致排序失败。了解了这一点，下面问题就简单多了，我们需要一棵多高的树才能保证其能够容纳N!个叶节点?<br>了解二叉树基本性质的人可以不假思索的回答，一棵高度为h的二叉树最多可以含有$2^h$个叶节点(此时人们称其为完全二叉树)，要让我们的排序算法万无一失，就必须保证 $2^h\geq N!$，即 $h(\text{比较次数})\geq log(N!)$。<br>到这里我们已经可以得出结论，任何基于元素间比较的排序算法复杂度不可能优于O(logN!)，可我还有一个问题，logN!到底是多大？阶乘计算总是过于复杂，人们一般采用<a href="https://zh.wikipedia.org/wiki/斯特靈公式">斯特灵公式</a>来求取一个近似值，遗憾的是我的数学能力仅停留在高中水平(高中数学还很少及格)，因此wiki上关于这段公式的内容在我眼里大概是这样的“~!@#$%^&amp;*#@^_^!”，好在<a href="http://stackoverflow.com/questions/2095395/is-logn-%CE%98n-logn">stackoverflow</a>上有一个更加简单的解法，Let&#39;s see:<br>$$\text{首先依据对数的性质可以得到：}log(N!)=log(N)+log(N-1)+log(N-2)+\ldots+log(N/2)+\ldots+log(1)$$$$\text{直接丢弃求和等式的右半部分可得：}\Rightarrow log(N!)\geq log(N)+log(N-1)+log(N-2)+\ldots+log(N/2)$$$$\Rightarrow log(N!)\geq log(N/2)+log(N/2)+log(N/2)+\ldots+log(N/2)$$$$\Rightarrow log(N!)\geq (N/2)log(N/2)=(N/2)logN-N/2$$$$\Rightarrow log(N!)= \Omega(NlogN)$$这里的$\Omega$指的是函数渐进增长的下界，可以依据类似的方式得到其上界：$$log(N!)=log(N)+log(N-1)+log(N-2)+\ldots+log(N/2)+\ldots+log(1)$$$$\Rightarrow log(N!)\leq log(N)+log(N)+log(N)+\ldots+log(N)=N\times log(N)$$$$\Rightarrow log(N!)=O(NlogN)$$<br>当既有$f(N)=O(g(N))$(upperbound)又有$f(N)=\Omega(g(N))$(lowerbound)时，我们说$f(N)=\Theta(g(N))$(tightbound)。在此例中即$log(N!)=\Theta(NlogN)$，如果你觉得这些关系很混乱，只要知道$\Theta$是O的充分不必要条件即可。</p>
<p><strong>结论：任何基于比较的排序算法，其最坏情形复杂度不可能优于$\Theta(NlogN)$</strong></p>
<h3 id="三-归并-Merge-排序"><a href="#三-归并-Merge-排序" class="headerlink" title="三.归并(Merge)排序"></a>三.归并(Merge)排序</h3><p>你可能以前从没听过归并排序，但你不可能没听过它的作者。现代计算机之父<a href="https://en.wikipedia.org/wiki/John_von_Neumann">约翰·冯·诺伊曼</a>于1945年发明归并排序，这被认为是最早在计算机上正确应用分治思想的算法。上码：<br><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">void</span> <span class="title function_">mergeSort</span><span class="params">(<span class="type">int</span>[] arr)</span>&#123;</span><br><span class="line">    <span class="type">int</span>[] aux=<span class="keyword">new</span> <span class="title class_">int</span>[arr.length];</span><br><span class="line">    mergeSort(arr,aux,<span class="number">0</span>,arr.length - <span class="number">1</span>);</span><br><span class="line">&#125;</span><br><span class="line"><span class="keyword">private</span> <span class="keyword">static</span> <span class="keyword">void</span> <span class="title function_">mergeSort</span><span class="params">(<span class="type">int</span>[] arr,<span class="type">int</span>[] aux,<span class="type">int</span> lo,<span class="type">int</span> hi)</span>&#123;</span><br><span class="line">    <span class="keyword">if</span>(lo&gt;=hi) <span class="keyword">return</span>;</span><br><span class="line">    <span class="type">int</span> mid=(lo+hi)&gt;&gt;<span class="number">1</span>;</span><br><span class="line"></span><br><span class="line">    <span class="comment">//divide</span></span><br><span class="line">    mergeSort(arr,aux,lo,mid);</span><br><span class="line">    mergeSort(arr,aux,mid+<span class="number">1</span>,hi);</span><br><span class="line"></span><br><span class="line">    <span class="comment">//merge</span></span><br><span class="line">    System.ArrayCopy(arr,lo,aux,lo,hi - lo + <span class="number">1</span>);</span><br><span class="line">    <span class="type">int</span> i=lo,j=mid+<span class="number">1</span>;</span><br><span class="line">    <span class="keyword">for</span>(<span class="type">int</span> k=lo;k&lt;=hi;k++)&#123;</span><br><span class="line">        <span class="keyword">if</span>(i&gt;mid) arr[k]=aux[j++];</span><br><span class="line">        <span class="keyword">else</span> <span class="keyword">if</span>(j&gt;hi) arr[k]=aux[i++];</span><br><span class="line">        <span class="keyword">else</span> <span class="keyword">if</span>(aux[i]&lt;aux[j]) arr[k]=aux[i++];</span><br><span class="line">        <span class="keyword">else</span> arr[k]=aux[j++];</span><br><span class="line">    &#125;</span><br><span class="line">    <span class="keyword">return</span>;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure></p>
<p>归并排序的思路是先找到输入数组的中间元素(mid)，然后根据中间元素将输入数组划分成两部分，整个过程递归进行，分到最后每个子数组将只有一个元素，此时所有的子数组就已经有序了(只有一个元素的数组是有序的)，然后对所有子数组执行归并(merge)操作，此时唯一需要做的就是保证归并完成后数组仍是有序的，如图：</p>
<div align=center><br><img src="/img/content/merge-sort.png" alt="merge-sort.png"><br></div>

<p>分治算法大多满足于同一个运行时间方程，即T(N)=2T(N/2)+O(N)，其中2T(N/2)代表求解两个同等规模的子问题需要的时间，O(N)是联合子问题的解需要的时间，对归并排序而言则指的是merge需要的时间，这里可以简单的将merge所需要的时间用N(实际是cN，这不影响结果)来代替。求解该方程是相当简单又不失趣味的过程：<br>$$T(N)=2T(N/2)+N \Rightarrow T(N)/N=T(N/2)/(N/2)+1$$$$T(N/2)=2T(N/4)+N/2 \Rightarrow T(N/2)/(N/2)=T(N/4)/(N/4)+1$$$$\vdots$$$$T(2)=2T(1)+1 \Rightarrow T(2)/2=T(1)+1$$将式子一层层代入后可得$T(N)/N=T(1)+logN\text{(等式右边一共有logN个1相加)}$，已知$T(1)=1$，最后等号两边同时乘以N得到:$$T(N)=N+N \times logN=O(NlogN)$$<br>以上证明过程假设了N是2的幂，即每次都能将输入数组分成两个正好同样大小的子数组，当输入数组是奇数的时候求解过程要麻烦一些，但结果是一样的。另外还可以采用类似的方法证明归并排序所需的比较次数最低为(NlogN)/2，最高为NlogN(注意这里没有O)，结合上一节的理论，可以总结一个非常拗口的结论，即不可能找到任何基于比较的排序算法，在最坏情况下所需比较次数少于归并排序在最坏情况下的比较次数。如果一条定理读起来非常拗口，往往意味着它没什么卵用，这里也是如此。<br>归并排序最令人诟病的缺陷在于它需要线性的空间复杂度，并且将元素从数组复制到辅助数组这个过程浪费了大量的时间。再加上考虑算法是否高效不能仅考虑最坏情形(还记得上篇文章插入排序与选择排序的例子吧？)，就平均速度而言它在高级排序算法中没有优势，种种因素导致归并排序使用场景非常有限，而这里我把它列为主流排序算法有三个原因：</p>
<ol>
<li>它是基于比较的排序算法中，所需比较次数最少的。</li>
<li>当空间不是问题的时候，人们有时出于稳定性(参考上篇文章“排序算法的稳定性”)考虑而选择归并排序，比如说JDK对引用类型使用归并排序。</li>
<li>它与快速排序设计思路类似，很适合放在快速排序之前讨论，就像上篇插入排序与希尔排序的关系一样。</li>
</ol>
<p><strong>结论：如果稳定性很重要而空间又不是问题，有时可以选择归并排序。</strong></p>
<h3 id="四-快速-Quick-排序"><a href="#四-快速-Quick-排序" class="headerlink" title="四.快速(Quick)排序"></a>四.快速(Quick)排序</h3><p>快速排序是目前使用最广泛的排序算法，它的作者是图灵奖得主<a href="https://en.wikipedia.org/wiki/Tony_Hoare">C.A.R.Hoare</a>，正如该算法的名字一样，它是实践中最快的已知排序算法，它在平均情况下的时间复杂度是O(NlogN)，这样看起来并没有什么特别之处，它之所以性能突出是因为其高度优化的内循环，换句话说，假定它的时间复杂度为k*NlogN+b，那么这里的k比其它线性对数级的算法要小。最原始的快速排序在最坏情形时复杂度为O(N^2)，但这完全可以通过一些编程技巧来避免。下面看代码：</p>
<figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">void</span> <span class="title function_">quickSort</span><span class="params">(<span class="type">int</span>[] arr)</span>&#123;</span><br><span class="line">    quickSort(arr,<span class="number">0</span>,arr.length - <span class="number">1</span>);</span><br><span class="line">&#125;</span><br><span class="line"><span class="keyword">private</span> <span class="keyword">static</span> <span class="keyword">void</span> <span class="title function_">quickSort</span><span class="params">(<span class="type">int</span>[] arr,<span class="type">int</span> lo,<span class="type">int</span> hi)</span>&#123;</span><br><span class="line">    <span class="keyword">if</span>(lo&gt;=hi) <span class="keyword">return</span>;</span><br><span class="line">    <span class="type">int</span> pivot=partition(arr,lo,hi); <span class="comment">//暂时先别管这条语句</span></span><br><span class="line">    quickSort(arr,lo,pivot - <span class="number">1</span>);</span><br><span class="line">    quickSort(arr,pivot + <span class="number">1</span>,hi);</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure>
<p>上面这段代码是否与归并排序如出一辙？因为它们都建立在分治思想之上，归并排序选取中点(mid)将数组均分成两个子数组，仔细思考一下，排序的merge过程建立在两个子数组已经有序的前提之下，那假设此时左子数组最大的元素小于右子数组最小的元素，是否可以完全去掉merge过程？举例来说假设一个数组的左子数组是[1,3]，右子数组是[5,6]，那么merge过程完全是多余的，因为该数组已经有序了。理解了这一点，你便理解了快速排序。剩下的问题是，如何找一个点切分数组，使其左子数组都小于等于该点，右子数组大于等于该点。<br>上面提到的这个点，我们一般称其为pivot(枢纽元)，而选取这个点的过程一般称其为partition(分区)，你可以完全不必理会这些专有名词，它们的主要作用在于吓唬那些不懂原理的人，让这些人对快速排序望而却步。但既然我们已经明白其原理了，这里我还是会使用它们来保持文章与主流书籍的一致性。下面先来看一种简单的分区思路：<br><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">/**</span></span><br><span class="line"><span class="comment">*直接选取第一个元素lo作为pivot，使用两个指针(i,j)扫描数组，i,j朝数组中间移动</span></span><br><span class="line"><span class="comment">*遇到arr[i]&gt;=pivot则i停止，遇到arr[j]&lt;=pivot则j停止，交换i,j对应的元素，重复该过程直到i,j相遇</span></span><br><span class="line"><span class="comment">*当整个过程停下来时将pivot与a[j]交换位置，从而完成分区</span></span><br><span class="line"><span class="comment">*</span></span><br><span class="line"><span class="comment">*       left part              center part           right part</span></span><br><span class="line"><span class="comment">* +--------------------------------------------------------------+</span></span><br><span class="line"><span class="comment">* lo |   &lt;= pivot   |                             |   &gt;= pivot   |</span></span><br><span class="line"><span class="comment">* +--------------------------------------------------------------+</span></span><br><span class="line"><span class="comment">* ^                 ^                             ^</span></span><br><span class="line"><span class="comment">* |                 |                             |</span></span><br><span class="line"><span class="comment">* pivot             i-&gt;                         &lt;-j</span></span><br><span class="line"><span class="comment">*</span></span><br><span class="line"><span class="comment">*/</span></span><br><span class="line"><span class="keyword">private</span> <span class="keyword">static</span> <span class="type">int</span> <span class="title function_">partition</span><span class="params">(<span class="type">int</span>[] arr,<span class="type">int</span> lo,<span class="type">int</span> hi)</span>&#123;</span><br><span class="line">    <span class="type">int</span> i=lo+<span class="number">1</span>,j=hi;</span><br><span class="line"></span><br><span class="line">    <span class="keyword">while</span>(<span class="literal">true</span>)&#123;</span><br><span class="line">        <span class="keyword">while</span>(arr[++i]&lt;arr[lo] &amp;&amp; i &lt; hi);</span><br><span class="line">        <span class="keyword">while</span>(arr[--j]&gt;arr[lo]); <span class="comment">//这里不需要判断j&gt;lo，因为当j=lo时，arr[j]&gt;arr[lo]不可能成立</span></span><br><span class="line">        <span class="keyword">if</span>(i&gt;=j) <span class="keyword">break</span>; 		 <span class="comment">//i,j相遇时停止</span></span><br><span class="line">        exchange(arr,i,j);</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    exchange(arr,lo,j);</span><br><span class="line">    <span class="keyword">return</span> j;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br>眼尖的同学一眼就能看出该程序的问题，考虑数组所有元素都等于pivot的情况，当i遇到等于pivot的元素时停下，j遇到等于pivot的元素也停下，然后i与j发生一次交换，这实际上什么也没干。了解这里为什么要多此一举，需要先分析算法的复杂度。<br>不难想象，快速排序的最好情况是每次选取的pivot正好在数组的中间，从而每次都把问题划分成相等规模的子问题，此时该算法的时间复杂度为T(N)=2T(N/2)+cN，cN表示分区过程消耗的时间，求解该方程的过程与刚才计算归并排序复杂度的过程一模一样，如果你记性还算不错的话应该知道结果是O(NlogN)。下面简单分析一下它的最坏情况，即每次选取的pivot都正好是数组最小的元素，此时划分的左子数组只有0个元素，右子数组为N-1个元素(去掉pivot)。其复杂度有如下关系：<br>$$T(N)=T(0)+T(N-1)+cN$$$$T(N-1)=T(0)+T(N-2)+c(N-1)$$$$T(N-2)=T(0)+T(N-3)+c(N-2)$$$$\vdots$$$$T(2)=T(0)+T(1)+c(2)$$<br>同样一顿代入后，可以得到$T(N)=(N-1)\times T(0)+T(1)+c(N\times(N-2)/2)$，其中c(N(N-2)/2)是由等差数列的求和公式而来。已知T(0)=T(1)=c，最后得到：<br>$$T(N)=c(N-1)+c+c(N(N-2)/2)=O(N^2)$$<br>在这种情况下，每次递归所做的仅仅是将pivot元素放到了正确位置了，这就跟选择排序一样低效。所以有人干脆选择在排序之前将数组随机打乱以降低发生这种事的可能性，Sedgewick在其书上写到，对于一个较大的数组，先将其随机打乱再进行快速排序，其时间复杂度是平方级的概率比你的电脑在排序时被闪电击中的概率要小得多。为此我还特意在网上查了一下电脑被闪电击中的概率，然而却没有得到满意的答案，因此我不太喜欢这种方式。后面你还会看到，大牛们发明了很多经过实践验证的优化手段，足以避免我们的电脑在排序的时候被闪电击中。<br>计算最佳与最坏情况的复杂度总是比较简单，而快速排序的平均复杂度又是如何得到的？我一直在盘算能不能通过一些写作技巧把这个问题糊弄过去，因为我查了很多资料，发现证明其平均复杂度的方式要么涉及概率学，要么涉及微积分，要么两个都涉及。前面我大概已经交代过我的数学水平了，总之，快速排序的平均复杂度是O(NlogN)，如果话说到这个份上你还要问我为什么，我想我们是没办法做朋友了。<br>咳，咳！还记得最开始的问题码？如果数组所有元素都等于pivot，i与j会不断发生无意义的交换，但最终j会停在数组的中间，而假设i与j遇到等于pivot的元素时都不停下，则j最终会停在数组的最左边，此时你将得到一个$O(N^2)$的排序算法。</p>
<p><strong>结论<br>快速排序是已知最快的排序算法，大多数场景都可以优先选择快速排序。而它的缺点是不稳定(注意这里的稳定指的不是“效率不稳定”，参见“排序算法的稳定性”)，运行效率在一定程度上依赖于概率(但通常比其它算法快)，实现它的内循环(partition)需要一定的编程技巧(但这也给编程高手留下了足够的优化空间)。
</strong></p>
<p>下篇文章来聊聊快速排序的优化历程。</p>
<p>本文参考链接</p>
<ul>
<li><a href="http://stackoverflow.com/questions/2095395/is-logn-%CE%98n-logn">is-logn-Θn-logn</a></li>
<li><a href="https://zh.wikipedia.org/wiki/%E4%BF%A1%E6%81%AF%E8%AE%BA">信息论</a></li>
<li><a href="http://algs4.cs.princeton.edu/23quicksort/">QUICKSORT</a></li>
</ul>
]]></content>
      
        <categories>
            
            <category> Algorithm </category>
            
        </categories>
        
        
        <tags>
            
            <tag> Sort </tag>
            
        </tags>
        
    </entry>
    
    <entry>
      <title><![CDATA[聊聊4种主流排序算法(上)]]></title>
      <url>https://blog.staynoob.cn/post/2016/04/%25E8%2581%258A%25E8%2581%258A4%25E7%25A7%258D%25E4%25B8%25BB%25E6%25B5%2581%25E6%258E%2592%25E5%25BA%258F%25E7%25AE%2597%25E6%25B3%2595-%25E4%25B8%258A/</url>
      <content type="html"><![CDATA[<blockquote>
<p>排序算法在很多计算机应用中都扮演着重要角色，同时也是大部分算法课程中不可或缺的一个章节。然而现如今的程序开发越来越模块化，程序员除了在技术面试的时候，几乎不存在任何应用场景需要手写一个排序算法，再则很多像我一样的学渣，看到“市面”上繁杂多样的排序算法就头疼，因此我打算在这里聊聊四种我个人认为比较主流的排序算法，看看当我们调用Arrays.sort()的时候到底发生了什么。</p>
</blockquote>
<h3 id="一-约定"><a href="#一-约定" class="headerlink" title="一.约定"></a>一.约定</h3><p>为了更加方便的表达意图，下面是关于本文的一些约定</p>
<ul>
<li>输入的规模<br>  文中使用N代表输入的规模，比如说N=100，表示将100个元素排序</li>
<li>对数<br>   除非有特别的说明，文中所有的对数都是以2为底的，即logN表示以2为底N的对数</li>
<li>代码<br>  文中的代码(java或js)大多数只截取关键片段，无法直接运行，关于这些算法的完整代码网上到处都是，这不是本文的核心，我相信文章使用尽可能少的代码对阅读者更友好，在文中写代码只能代表我无法用简短的人类语言来描述这部分意图。</li>
</ul>
<span id="more"></span>
<h3 id="二-简单的概念"><a href="#二-简单的概念" class="headerlink" title="二.简单的概念"></a>二.简单的概念</h3><p>在正式开始前先简单介绍几个基本概念（有经验的程序员请一定跳过这个部分）</p>
<ul>
<li>排序算法的稳定性<br>  我刚开始学编程的时候看到“不稳定的算法”，首先想到的是“是不是这种算法有时无法将数组恢复有序？或者说有时运行非常快，有时永远也得不出结果？”事实上上面两种情况只能称作有bug的算法，算法的稳定性指的是当有两个元素有相同的主键的时候，算法是否能保证在排序前后这两个元素的相对位置不变，比如有一个数组  <figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[&#123;<span class="attr">id</span>:<span class="number">1</span>,<span class="attr">name</span>:<span class="string">&#x27;foo&#x27;</span>&#125;,&#123;<span class="attr">id</span>:<span class="number">1</span>,<span class="attr">name</span>:<span class="string">&#x27;bar&#x27;</span>&#125;]</span><br></pre></td></tr></table></figure>
  当依据id对这个数组排序后，是否还能保证foo在bar的前面。如果可以，则该排序算法是稳定的。</li>
<li>算法分析<br>  我们应该怎样来判断一个算法，或者说一段代码的性能？最简单的方法通过反复测试来鉴别，但这显然太没效率了。我们需要在算法的设计阶段就能大致预测出算法能跑多块。因此高德纳(D.E. Knuth)首先提出构造数学模型来描述任意程序的运行时间，简单来说假设计算机执行每条语句的耗时是一个常数C，假设程序需要执行n条语句才能完成计算，则程序的总运行时间是C*n。常数C往往跟计算机硬件，使用的编程语言(PHP是世界上最好的语言！)，CPU资源分配等相关，因此我们只需要关心n就可以了，而影响n的最关键因素是算法中的循环与递归。举个简单的例子:  <figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="type">int</span> N=arr.length;</span><br><span class="line"><span class="keyword">for</span>(<span class="type">int</span> i=<span class="number">0</span>;i &lt; N;i++)&#123;</span><br><span class="line">    <span class="keyword">for</span>(<span class="type">int</span> j=i+<span class="number">1</span>;j &lt; N;j++)&#123;</span><br><span class="line">    	<span class="keyword">if</span>(arr[i]+arr[j]==<span class="number">0</span>) <span class="keyword">return</span> <span class="literal">true</span>;</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br><span class="line"><span class="keyword">return</span> <span class="literal">false</span>;</span><br></pre></td></tr></table></figure>
  上面这段代码搜索给定数组中是否包含两个和为零的元素，其中执行次数最多的是处于内循环中的判断语句，最坏的情况下要执行N*(N-1)次，当N足够大时，N*(N-1)≈N^2。于是我们可以说上面这段代码在最坏情况下运行时间大致为 C*(N^2)，我们把这称为最坏情况下的时间复杂度。<br>  同时，算法分析还应该考虑运行算法需要多少额外的存储空间，即空间复杂度，但由于空间复杂度往往很容易计算，因此不必在上面花太多时间。<br>  顺带提一下，上面这段代码很常见，但不适用于大规模输入的情况，学过算法的同学应该知道，可以通过运行一个二分查找使得最坏情况下运行时间为C*(N*logN)，还可以通过Hash算法将其优化到C*N</li>
<li><p>时间复杂度<br>  大部分情况使用O(N),即“大O记法”来表示算法运行时间增长的上界，比如1000*N=O(N^2),代表随着N的增大，1000*N的增长速度总是小于或等于N^2的增长速度。严格的定义如下:</p>
<blockquote>
<p>如果存在正常数c和$n_0$,使得当N&gt;=$n_0$时T(N)&lt;=cf(N),则记为T(N)=O(f(N))。</p>
</blockquote>
<p>  使用大O计法来表示前面那段程序在最坏情况的时间复杂度就是 C*(N^2)=O(N^2)。另外一个重要结论是:</p>
<blockquote>
<p>当N越来越大的时候，logN增大得非常慢，N*logN还可以接受，N^2比前面两者要大得多。</p>
</blockquote>
<p>  如果以上看起来很复杂，你只需要像我一样知道O(NlogN)比O(N^2)好就可以了。<br>  偶尔也可能使用Ω(Omega),θ(theta)等符号，感兴趣的话可以去网上查找这些符号的定义，不知道也关系不大。</p>
</li>
</ul>
<h3 id="三-插入-Insertion-排序"><a href="#三-插入-Insertion-排序" class="headerlink" title="三.插入(Insertion)排序"></a>三.插入(Insertion)排序</h3><p>插入排序是唯一一种被广泛应用的平方级(quodradic)排序算法，大量编程语言基础库(STL,JDK等)都能见到它矫健的身影，下面先上代码：<br><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">for</span>(<span class="type">int</span> i=<span class="number">1</span>;i &lt; arr.length;i++)&#123;</span><br><span class="line">    <span class="keyword">for</span>(<span class="type">int</span> j=i;j &gt; <span class="number">0</span> &amp;&amp; arr[j] &lt; arr[j-<span class="number">1</span>];j--)&#123;</span><br><span class="line">        exchange(j,j-<span class="number">1</span>);</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br>插入排序的原理有些类似于打扑克牌抓牌的过程，脑补一下你在玩<strong>升级</strong>这类扑克牌游戏时抓牌的过程，你就学会了插入排序，唯一的区别是当你新抓一张牌时，你一眼就能找到要插入的位置，而计算机需要一定次数的比较才能找到。数组如果一开始是逆序，则插入排序每抓一张牌都要将其跟手上每一张牌进行比较，从而找到正确的插入位置。这就是插入排序的最坏情况。从中很容易得出，插入排序的时间复杂度是O(N^2)，而你“抓牌”的时间复杂度是O(N)。<br>你可能还听过几种平方级算法，比如说选择排序和冒泡排序，它们的应用场景少的可怜。大多数时候它们仅仅用来教计算机初学者如何写代码，国外的一些视频教程甚至直接称之为“玩具”或者“垃圾”算法。你可能会觉得奇怪，为什么同为平方级算法，插入排序如此流行，而选择排序的意义跟<strong>HelloWorld</strong>差不多，先上一张图</p>
<div align=center><br><img src="/img/content/insertion-vs-selection.png" alt="insertion-vs-selection"><br></div>

<p>上图描述了插入排序与选择排序的区别，可以看到在平均情况下，插入排序需要进行的比较次数比选择排序几乎少一半。插入排序只有在最坏的情况(输入数组是逆序)下时间复杂度是O(N^2)，在最好的情况(数组已经有序)下是O(N)，而冒泡排序与选择排序无论在什么情况下都是O(N^2)。这个例子也可以说明，比较两种算法，不能仅仅看它们在最差情况下的时间复杂度。<br>以上说明了如果我们只能选择平方级算法，应该优先考虑插入排序。而大牛们早已证明了最优的排序算法(基于比较的)复杂度是O(N*logN)。那又是什么让插入排序得以与这些高级算法竞争呢？<br>后面你会看到，O(N*logN)排序算法大多基于分治(Divide and Conquer)思想，而该思想的实现大多基于递归，在大多数编程语言中，调用函数的开销是非常大的，需要为每个函数分配栈空间，有时还需要保存其作用域直到整个算法结束，这个过程稍有不慎就会得到一个StackOverFlow(栈空间溢出)，部分函数式编程语言使用<a href="https://en.wikipedia.org/wiki/Tail_call">尾递归优化(Tail Call Optimization)</a>来缓解这类问题。总之在大多数情况下，函数调用语句无论是时间还是空间花费都高于执行普通语句。于是基于递归的高级排序算法在处理小型数组时性能甚至低于插入排序。</p>
<p><strong>结论：插入排序适用于初始状态就接近有序的大型数组，此时它的复杂度几乎是线性的，但它更加主要的应用场景是处理微型数组。大量基于递归的排序算法，在递归函数调用前判断当前需要处理的数组大小，如果输入小于某个常量则直接执行插入排序，至于这个常量的大小则因使用环境而异，有人说大概5-15，有人说5-20</strong></p>
<h3 id="四-希尔-Shell-排序"><a href="#四-希尔-Shell-排序" class="headerlink" title="四.希尔(Shell)排序"></a>四.希尔(Shell)排序</h3><p>前面提到，插入排序在处理大规模随机数组时是非常低效的，它最主要的缺陷在于每次只交换相邻的两个元素，元素只能一步一步的从数组的一端移动到另一端。Donald Shell在此基础上，对其进行了一个小小的改进，于是有了以他名字命名的希尔排序。希尔排序是第一批突破平方级别的算法之一，然而理解它的性能至今仍是一项挑战。上代码：<br><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="type">int</span> N=arr.length;</span><br><span class="line"><span class="type">int</span> step=<span class="number">1</span>;</span><br><span class="line"><span class="keyword">while</span>(step &lt; arr.length/<span class="number">3</span>) step=<span class="number">3</span>*step+<span class="number">1</span>; <span class="comment">//&#123;1,4,7,10...&#125;，注意这个序列有更好的选择，详见下文</span></span><br><span class="line"><span class="keyword">while</span>(step &gt;= <span class="number">1</span>)&#123;</span><br><span class="line">    <span class="keyword">for</span>(<span class="type">int</span> i=step;i &lt; arr.length;i++)&#123;</span><br><span class="line">        <span class="keyword">for</span>(<span class="type">int</span> j=i;j &gt; i-step &amp;&amp; arr[j] &lt; arr[j-step];j-=step)&#123;</span><br><span class="line">            exchange(j,j-<span class="number">1</span>);</span><br><span class="line">        &#125;</span><br><span class="line">    &#125;</span><br><span class="line">	step/=<span class="number">3</span>;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><br>希尔排序与选择排序最主要的差别在于引入了一个step变量(指步长，一些书上喜欢用gap或h来表示)，这使得每次交换元素位置，都可以使该元素向其最终位置跨一大步，看图：</p>
<div align=center><br><img src="/img/content/insertion-vs-shell.png" alt="insertion-vs-shell"><br></div>

<p>随着排序的进行，数组越来越接近有序，步长也越来越小，直到step=1，此时希尔排序就变得跟插入排序一模一样了，但此时数组已经几乎完全有序了，还记得前面所说过的吗？对一个几乎有序的数组运行插入排序，其复杂度接近O(N)。整个过程看起来天衣无缝，然而其中隐藏着一个难点，应该使用怎样的步长序列？必须要考虑的因素有两点：</p>
<ul>
<li>当改变步长的时候，如何保证新的步长不会打乱之前排序的结果？这不会影响最终排序的正确性，因为只要步长在减小，数组永远都只会朝着更加有序的方向迈进，但这却是影响希尔排序效率的关键。因为这涉及到完成排序的过程中，算法做了多少无用功。</li>
<li>如何保证每一个步长都是有意义的？来看一个例子，假设有一个数组[1,5,2,6,3,7,4,8]，使用步长序列[4,2,1]对其进行排序，过程如图：<br><div align=center><br><img src="/img/content/shell-sort.png" alt="shell-sort.png"><br></div><br>这就相当于进行了一次低效的插入排序，因为在step=1之前，程序什么也没干，偶数位置永远不会与基数位置进行比较。而下文你会看到，[4,2,1]正是按照Shell本人一开始推荐的增量算法得来的。</li>
</ul>
<p>以上应该足以说明寻找好的步长序列是一件非常困难的事。好在总有人愿意为我们做这件事，下面列出一些流行的增量序列：</p>
<ol>
<li>Shell增量:N/2,N/4,N/8...1<br> 即初始步长选择N/2，后面每次取半直到步长为1，它出自Shell本人且非常容易用代码表达，因此而流行，我看到现在的一些文章都还在使用它。然而前面的例子已经可以看出，在一些特定输入下它非常低效，它在最坏情形的时间复杂度仍是O(N^2)。事实上除了这个序列，后面列出的序列在最坏情形的性能差距都不会很大。</li>
<li>Hibbard增量:1,3,7,15...,2^k-1<br> 最坏情形$\theta(N^{3/2}))$,平均复杂度大概是$O(N^{5/4})$，它的重要改进是相邻的增量之间没有公因子。而Shell增量之所以不好，因为它的增量之间并非互素，如果这很难理解就回顾一下上面使用[4,2,1]步长排序的例子，再脑补一下用[6,3,1]步长对[1,5,9,2,6,10,3,7,11,4,8,2]排序的情形。</li>
<li>Knuth增量:1,4,13,40,...,(3^k - 1)/2<br> 同为$\theta(N^{3/2})$，但貌似这个序列使用最广泛，即便现在已经证实了有更好的步长序列。</li>
<li>Sedgewick增量:1,5,19,41,109...<br> 最坏情形$O(N^{4/3})$，这是已知复杂度(还有一些实践起来效果很好的序列至今没有算出复杂度)的最佳序列，通过<br> $$1,19,109,505,...,9\times4^k-9\times2^k+1$$$$5,41,209,929,...,2^{k+2}\times(2^{k+2}-3)+1$$<br> 两个算式综合而来，依据基偶性交叉选择两个算式的结果即可。<br>关于希尔排序的研究仍在继续，目前还没有人证明某个序列是最优的，理解希尔排序在不同的序列下的复杂度是非常高端的学问。好在实现它是非常简单的。把大牛们的工作结晶放在一个常量数组中就可以很容易的实现希尔排序。</li>
</ol>
<p><strong>结论：希尔排序处理中等大小的数组是非常不错的，对于非常大的输入也可以在有限的时间内完成。它的优势在于编程简单，无需额外空间。Sedgewick在其书中的原话是“如果你需要解决一个排序问题，有没有系统排序函数可用(例如直接接触硬件或是运行嵌入式系统中的代码)，可以先用希尔排序，然后再考虑是否值得将它替换成更加复杂的排序算法。”
</strong></p>
<p>下一篇文章来聊剩下两个算法。</p>
<p>本文参考链接</p>
<ul>
<li><a href="http://algs4.cs.princeton.edu/21elementary/">Elementary Sorts</a></li>
<li><a href="http://www.stoimen.com/blog/2012/02/27/computer-algorithms-shell-sort/">Computer Algorithms: Shell Sort</a></li>
<li><a href="http://faculty.simpson.edu/lydia.sinapova/www/cmsc250/LN250_Weiss/L12-ShellSort.htm#increments">Sorting Algorithms - Shellsort</a></li>
<li><a href="http://www.cs.princeton.edu/~rs/shell/paperF.pdf">Analysis of Shellsort and Related Algorithms</a></li>
</ul>
]]></content>
      
        <categories>
            
            <category> Algorithm </category>
            
        </categories>
        
        
        <tags>
            
            <tag> Sort </tag>
            
        </tags>
        
    </entry>
    
    <entry>
      <title><![CDATA[安装ubuntu16.04后要做的9件事]]></title>
      <url>https://blog.staynoob.cn/post/2016/04/%25E5%25AE%2589%25E8%25A3%2585ubuntu16-04%25E5%2590%258E%25E8%25A6%2581%25E5%2581%259A%25E7%259A%25849%25E4%25BB%25B6%25E4%25BA%258B/</url>
      <content type="html"><![CDATA[<blockquote>
<p>ubuntu在4月21日发布了16.04LTS版本,网上已经有大家疯狂转载的类似标题的文章，这里记录适用于我个人的要做的9件事(目前我电脑还安装了win10),</p>
</blockquote>
<h3 id="一-将CAPSLOCK改为ctrl键"><a href="#一-将CAPSLOCK改为ctrl键" class="headerlink" title="一.将CAPSLOCK改为ctrl键"></a>一.将CAPSLOCK改为ctrl键</h3><p>按<code>alt+F2</code>，搜索gnome-session-properties，添加启动命令<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">setxkbmap -option ctrl:nocaps</span><br></pre></td></tr></table></figure><br>谁用谁知道</p>
<h3 id="二-修改root密码-取消sudo密码"><a href="#二-修改root密码-取消sudo密码" class="headerlink" title="二.修改root密码,取消sudo密码"></a>二.修改root密码,取消sudo密码</h3><p>新安装的linux系统默认是没有启用root账户的,需要使用sudo命令修改root密码,同时作为桌面版系统,每次使用sudo命令都需要输入密码显得很多余<br>使用如下命令修改root密码<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">sudo passwd root</span><br></pre></td></tr></table></figure></p>
<p>使用如下命令取消sudo密码<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">sudo visudo</span><br></pre></td></tr></table></figure><br>该命令实际作用是打开编辑器编辑 <strong>/etc/sudoers</strong> 里面的内容,只是在修改完成后会自动检查格式是否有效,这里只要修改sudo用户组,添加 <strong>NOPASSWD</strong> 即可<br><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">#Allow members of group sudo to execute any command</span><br><span class="line">%sudo ALL=(ALL:ALL) NOPASSWD:ALL</span><br></pre></td></tr></table></figure></p>
<span id="more"></span>
<h3 id="三-修改grub引导等待时间"><a href="#三-修改grub引导等待时间" class="headerlink" title="三.修改grub引导等待时间"></a>三.修改grub引导等待时间</h3><p>grub的配置文件在 <strong>/boot/grub/grub.cfg</strong>,但该文件实际上是根据 <strong>/etc/default/grub</strong> 的配置自动生成的,So<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">sudo vim /etc/default/grub</span><br></pre></td></tr></table></figure><br>注释掉 <strong>GRUB_HIDDEN_TIMEOUT=0</strong> 修改 <strong>GRUB_TIMEOUT=3</strong> 这里的3指的是等待3秒的意思,Then<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">sudo update-grub</span><br></pre></td></tr></table></figure><br>即重新生成grub的配置文件</p>
<h3 id="四-禁止开机时的系统错误弹窗"><a href="#四-禁止开机时的系统错误弹窗" class="headerlink" title="四.禁止开机时的系统错误弹窗"></a>四.禁止开机时的系统错误弹窗</h3><p>ubuntu系统的老问题了,只要 <strong>/var/crash/</strong> 目录中有文件存在,则每次开机都会弹窗“检查到系统程序错误”,可以通过 <code>rm /var/crash/*</code> 暂时解决这个问题,但每当有新的 &quot;crash&quot; 文件生成,则该问题重现,推荐的办法是<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">sudo vim /etc/default/apport</span><br></pre></td></tr></table></figure><br>将其中的 <strong>enabled</strong> 属性由1改成0</p>
<h3 id="五-将用户目录的文件夹改回英文"><a href="#五-将用户目录的文件夹改回英文" class="headerlink" title="五.将用户目录的文件夹改回英文"></a>五.将用户目录的文件夹改回英文</h3><p>ubuntu安装语言选择中文,自动生成的目录也成了中文,导致命令行操作非常麻烦,打开终端输入<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="built_in">export</span> LANG=en_US</span><br><span class="line">xdg-user-dirs-gtk-update</span><br><span class="line"><span class="built_in">export</span> LANG=zh-CN</span><br></pre></td></tr></table></figure><br>在弹出的窗口中选择将文件夹改成英文,在下次启动时选择不改成中文并且不再提示即可</p>
<h3 id="六-梯子"><a href="#六-梯子" class="headerlink" title="六.梯子"></a>六.梯子</h3><ol>
<li><a href="https://github.com/shadowsocks/shadowsocks-qt5">shadowsocks-qt5</a> <figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">sudo add-apt-repository ppa:hzwhuang/ss-qt5</span><br><span class="line">sudo apt-get update</span><br><span class="line">sudo apt-get install shadowsocks-qt5</span><br></pre></td></tr></table></figure>
 将命令<code>/usr/bin/ss-qt5</code>添加至开机启动</li>
<li><a href="https://github.com/getlantern/lantern">lantern</a><br> 将如下命令添加至开机启动 <figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">/usr/bin/nohup sh -c <span class="string">&quot;/bin/sleep 3 &amp;&amp; /usr/lib/lantern/lantern.sh -addr 0.0.0.0:8787 -startup=true &amp;&gt; /dev/null&quot;</span> &amp;&gt; /dev/null</span><br></pre></td></tr></table></figure>
</li>
</ol>
<h3 id="七-设置系统时间为本地时间-单操作系统可忽略"><a href="#七-设置系统时间为本地时间-单操作系统可忽略" class="headerlink" title="七.设置系统时间为本地时间(单操作系统可忽略)"></a>七.设置系统时间为本地时间(单操作系统可忽略)</h3><p>linux系统向来将BIOS时间视为GMT标准时间,于是将当前BIOS时间+当前时区的时差作为当前系统时间,而windows则直接使用BIOS时间作为当前系统时间,且两个系统都会在关机时将时间回写到BIOS上,这就造成了两个系统之间永远都有8小时(以北京时间计算)的时差,随便修改哪个系统都可以解决这个问题,这里只说linux解决方案,在16.04以前的版本中,可以修改 <strong>/etc/default/rcS</strong> 将里面的 <strong>UTC=yes</strong> 改成 <strong>no</strong> 即可,而ubuntu16.04无法再通过该文件配置,现在应该运行如下命令<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">sudo timedatectl set-local-rtc 1</span><br></pre></td></tr></table></figure></p>
<h3 id="八-开机自动挂载ntfs分区-单操作系统可忽略"><a href="#八-开机自动挂载ntfs分区-单操作系统可忽略" class="headerlink" title="八.开机自动挂载ntfs分区(单操作系统可忽略)"></a>八.开机自动挂载ntfs分区(单操作系统可忽略)</h3><p>ubuntu只会自动挂载安装时分配给ubuntu的分区,导致每次开机后都要手动mount其它分区<br>可以修改 <strong>/etc/fstab</strong> 按照该文件的格式添加要挂载的分区即可,UUID可以通过 <code>blkid</code> 命令获取,文件系统格式一般写 auto 就可以了</p>
<h3 id="九-搭建开发环境-非码农可忽略"><a href="#九-搭建开发环境-非码农可忽略" class="headerlink" title="九.搭建开发环境(非码农可忽略)"></a>九.搭建开发环境(非码农可忽略)</h3><ul>
<li>安装java  <figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">sudo add-apt-repository ppa:webupd8team/java</span><br><span class="line">sudo apt update</span><br><span class="line">sudo apt install oracle-java8-installer</span><br></pre></td></tr></table></figure></li>
<li>安装node<br>  推荐使用nvm安装node,将以下命令中 &quot;$HOME/.nvm&quot; 替换成你想要的安装路径即可  <figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.31.0/install.sh | NVM_DIR=<span class="string">&quot;<span class="variable">$HOME</span>/.nvm&quot;</span> bash</span><br></pre></td></tr></table></figure>
  重新打开终端,输入以下命令安装任意版本的nodejs  <figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">nvm install &lt;version&gt;</span><br></pre></td></tr></table></figure>
  如果在使用“局域网”，考虑将npm切换至淘宝镜像(下面的命令等价于修改~/.npmrc)  <figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">npm config <span class="built_in">set</span> registry <span class="string">&quot;https://registry.npm.taobao.org&quot;</span></span><br><span class="line">npm config <span class="built_in">set</span> phantomjs_cdnurl <span class="string">&quot;http://cnpmjs.org/downloads&quot;</span></span><br><span class="line">npm config <span class="built_in">set</span> sass_binary_site <span class="string">&quot;https://npm.taobao.org/mirrors/node-sass/&quot;</span></span><br></pre></td></tr></table></figure>
  使用nvm管理node版本的好处是，当需要升级node版本时，可以直接使用如下命令自动安装全局模块  <figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">nvm install NEW_VERSION --reinstall-packages-from=OLD_VERSION</span><br></pre></td></tr></table></figure></li>
<li>安装mysql  <figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">sudo apt install mysql-server mysql-client</span><br></pre></td></tr></table></figure>
  之后修改/etc/mysql/my.cnf，添加如下配置节修改默认的字符集  <figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">[client]</span><br><span class="line">default-character-set=utf8</span><br><span class="line">[mysql]</span><br><span class="line">default-character-set=utf8</span><br><span class="line">[mysqld]</span><br><span class="line">collation-server = utf8_unicode_ci</span><br><span class="line">init-connect=&#x27;SET NAMES utf8&#x27;</span><br><span class="line">character-set-server = utf8</span><br></pre></td></tr></table></figure></li>
<li>安装mongodb<br>  可以直接使用apt命令来安装mongodb，但建议使用mongodb官方的仓库来安装最新版本（当前版本为3.4），安装过程要稍微麻烦一些，首先需要导入GPGkey  <figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 0C49F3730359A14518585931BC711F9BA15703C6</span><br></pre></td></tr></table></figure>
  添加软件源  <figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="built_in">echo</span> <span class="string">&quot;deb [ arch=amd64,arm64 ] http://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/3.4 multiverse&quot;</span> | sudo <span class="built_in">tee</span> /etc/apt/sources.list.d/mongodb-org-3.4.list</span><br></pre></td></tr></table></figure>
  之后就可以直接使用apt命令安装了  <figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">sudo apt update &amp;&amp; sudo apt install mongodb-org</span><br></pre></td></tr></table></figure>
  如果你安装的不是最新版，可能还需要手动添加<em>service unit</em>，类似下面这样  <figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line">[Unit]</span><br><span class="line">Description=High-performance, schema-free document-oriented database</span><br><span class="line">After=network.target</span><br><span class="line"></span><br><span class="line">[Service]</span><br><span class="line">User=mongodb</span><br><span class="line">ExecStart=/usr/bin/mongod --quiet --config /etc/mongod.conf</span><br><span class="line"></span><br><span class="line">[Install]</span><br><span class="line">WantedBy=multi-user.target</span><br></pre></td></tr></table></figure>
</li>
</ul>
<p>return 0;</p>
]]></content>
      
        <categories>
            
            <category> System </category>
            
        </categories>
        
        
        <tags>
            
            <tag> Linux </tag>
            
        </tags>
        
    </entry>
    
    <entry>
      <title><![CDATA[init commit]]></title>
      <url>https://blog.staynoob.cn/post/2016/04/init-commit/</url>
      <content type="html"><![CDATA[<!-- ![init commit](/img/init_commit.jpg) -->
<blockquote>
<p>Sometimes I talk to myself because I need expert advice.</p>
</blockquote>
<p>OK!Let&#39;s begin.<br>//以上是这篇文章的开头</p>
<span id="more"></span>
<p>快问我,为什么有一个这么中二的开头!<br>学写代码已经一年多了!这一年最大的成果就是,自己已经越来越不会与人交流了,于是在不断的码字,然后删除的循环之下,诞生了这个毫无意义的段落</p>
<p>//So,我搭建了这个博客,希望自己还能够写出人类能够理解的文字,在这里我会发表自己对一些技术的看法<br>//以及尽可能多的谈一些与技术无关的话题<br>我搭建了这个博客,并且打算在这里瞎扯淡</p>
<p>return 0;</p>
]]></content>
      
        <categories>
            
            <category> Diary </category>
            
        </categories>
        
        
    </entry>
    
  
  
</search>