2019-01-10 02:43:55	UTF-8 Char Found In TRB Genesis	[Forward]<br /><br />This blog post is a <a href="http://mod6.net/2018/March/12/genesis-regrind/index.html">cut-down copy</a> from the my original <a href="http://mod6.net/blog.html">blog spot</a>.   First, I wanted to repost it as I make use of the tool contained in here called 'check_nonascii_bytes.pl' during my TRB Keccak Regrind.  Second, I wanted to cut out the latter part of the original post; at the time, I had tak,en the extra steps to regrind vpatches that were impacted by the single change to the genesis.vpatch; however, since TRB was getting a full regrind with keccak hashes, it's no longer relevant to repost the old experimental vpatches.<br /><br />[ Preface ]<br /><br />It was brought to the attention of The Bitcoin Foundation by <a href="http://wot.deedbot.org/BDDE12104FE81BE7F83B698F5356DE4752432A9E.html">phf</a> that a UTF-8 character was <a href="http://btcbase.org/log/2018-02-25#1786028">discovered</a> in the original <a href="http://thebitcoin.foundation/v/patches/genesis.vpatch">genesis.vpatch</a> of The Bitcoin Reference Implementation, also known as The Real Bitcoin (TRB). UTF-8 characters are forbidden in The Reference Implementation source code, and will be removed. This post meant to present, in detail for Lords of the Most Serene Republic, all of the requisite work to remove the UTF-8 character, and regrind <a href="http://thebitcoin.foundation/v/patches/genesis.vpatch">genesis.vpatch</a> as well as descendant vpatches in v0.5.4 RELEASE.<br /><br />[ The Offending Character ]<br /><br />Upon <a href="http://btcbase.org/log/2018-02-25#1786094">inspection</a> of we have found the <a href="http://archive.is/1w3Kx">original</a> culprit of this offense. The character in question appears on line 23962 of <a href="http://thebitcoin.foundation/v/patches/genesis.vpatch">genesis.vpatch</a>. It is a substituted Unicode character for a hyphen.<br /><br />[ First Things First: Seek & Destroy ]<br /><br />After the UTF-8 character was found in <a href="http://thebitcoin.foundation/v/patches/genesis.vpatch">genesis.vpatch</a> I took it upon myself to search through the rest of the v0.5.4 Release vpatches to see if any others were lurking in the thousands of lines of source code.<br /><br />There are two ways to go about the ``Seek & Destroy UTF-8'' mission: Automate the search, and do the check by hand. To cover the former, automation, I created two different programs, one in perl, and one in C to check each character and see if it fell within the 7-Bit ASCII range. It should be said that a check-by-hand is also in necessary, as even the most well intentioned automation can miss something that would be obvious to the human eye; However, to save us time right now, (there are well over a million bytes to examine contained in v0.5.4 Release) we'll just check with automation.<br /><br /><pre><br />mod6@gentoo ~/check_nonascii_bytes $ ls patches/<br />asciilifeform-kills-integer-retardation.vpatch                         bitcoin-asciilifeform.2-https_snipsnip.vpatch<br />asciilifeform_add_verifyall_option.vpatch                              bitcoin-asciilifeform.3-turdmeister-alert-snip.vpatch<br />asciilifeform_and_now_we_have_block_dumper_corrected.vpatch            bitcoin-asciilifeform.4-goodbye-win32.vpatch<br />asciilifeform_and_now_we_have_eatblock.vpatch                          bitcoin-v0_5_3-db_config.6.vpatch<br />asciilifeform_dns_thermonyukyoolar_kleansing.vpatch                    bitcoin-v0_5_3_1-rev_bump.7.vpatch<br />asciilifeform_dnsseed_snipsnip.vpatch                                  bitcoin-v0_5_3_1-static_makefile_v002.8.vpatch<br />asciilifeform_lets_lose_testnet.vpatch                                 genesis.vpatch<br />asciilifeform_maxint_locks_corrected.vpatch                            makefiles.vpatch<br />asciilifeform_orphanage_thermonuke.vpatch                              malleus_mikehearnificarum.vpatch<br />asciilifeform_tx-orphanage_amputation.vpatch                           mod6_der_high_low_s.vpatch<br />asciilifeform_ver_now_5_4_and_irc_is_gone_and_now_must_give_ip.vpatch  mod6_fix_dumpblock_params.vpatch<br />asciilifeform_zap_hardcoded_seeds.vpatch                               programmable-versionstring.vpatch<br />asciilifeform_zap_showmyip_crud.vpatch                                 rm_rf_upnp.vpatch<br />bitcoin-asciilifeform.1.vpatch<br />mod6@gentoo ~/check_nonascii_bytes $ ls -l patches | awk '{print $5}' | perl -e '@a=<STDIN>; $total=0; foreach $n (@a) { chomp($n); $total+=$n; } print "Total Bytes: $totaln";'<br />Total Bytes: 1049853<br /> </pre><br /><br />WARNING: Neither of these programs are pretty, or meant to be used for anything else, but for our purposes here: They get the job done.<br /><br />Let's start with the perl program that I wrote to search through all the v0.5.4 Release vpatches. First, I'll present the execution of this program, followed by posting the source code.<br /><br /><pre><br />mod6@gentoo ~/check_nonascii_bytes $ ./v.pl i http://thebitcoin.foundation <br />Full vpatch sync complete to "/home/mod6/check_nonascii_bytes/patches"<br />Seal sync complete to "/home/mod6/check_nonascii_bytes/.seals"<br />mod6@gentoo ~/check_nonascii_bytes $ for i in `ls patches`; do echo "vpatch: $i" && cat patches/$i | ./check_nonascii_bytes.pl; done  <br />vpatch: asciilifeform-kills-integer-retardation.vpatch<br />vpatch: asciilifeform_add_verifyall_option.vpatch<br />vpatch: asciilifeform_and_now_we_have_block_dumper_corrected.vpatch<br />vpatch: asciilifeform_and_now_we_have_eatblock.vpatch<br />vpatch: asciilifeform_dns_thermonyukyoolar_kleansing.vpatch<br />vpatch: asciilifeform_dnsseed_snipsnip.vpatch<br />vpatch: asciilifeform_lets_lose_testnet.vpatch<br />vpatch: asciilifeform_maxint_locks_corrected.vpatch<br />vpatch: asciilifeform_orphanage_thermonuke.vpatch<br />vpatch: asciilifeform_tx-orphanage_amputation.vpatch<br />vpatch: asciilifeform_ver_now_5_4_and_irc_is_gone_and_now_must_give_ip.vpatch<br />vpatch: asciilifeform_zap_hardcoded_seeds.vpatch<br />vpatch: asciilifeform_zap_showmyip_crud.vpatch<br />vpatch: bitcoin-asciilifeform.1.vpatch<br />vpatch: bitcoin-asciilifeform.2-https_snipsnip.vpatch<br />vpatch: bitcoin-asciilifeform.3-turdmeister-alert-snip.vpatch<br />vpatch: bitcoin-asciilifeform.4-goodbye-win32.vpatch<br />vpatch: bitcoin-v0_5_3-db_config.6.vpatch<br />vpatch: bitcoin-v0_5_3_1-rev_bump.7.vpatch<br />vpatch: bitcoin-v0_5_3_1-static_makefile_v002.8.vpatch<br />vpatch: genesis.vpatch<br />NON-7BIT-ASCII VALUE IN SOURCE FILE: a/bitcoin/src/util.h ; BYTE VALUE: e2<br />NON-7BIT-ASCII VALUE IN SOURCE FILE: a/bitcoin/src/util.h ; BYTE VALUE: 80<br />NON-7BIT-ASCII VALUE IN SOURCE FILE: a/bitcoin/src/util.h ; BYTE VALUE: 94<br />vpatch: makefiles.vpatch<br />vpatch: malleus_mikehearnificarum.vpatch<br />vpatch: mod6_der_high_low_s.vpatch<br />vpatch: mod6_fix_dumpblock_params.vpatch<br />vpatch: programmable-versionstring.vpatch<br />vpatch: rm_rf_upnp.vpatch<br />mod6@gentoo ~/check_nonascii_bytes $<br /></pre> <br /><br />As we can see from the above, we sync'd all of the vpatches and seals from The Bitcoin Foundation's mirror of v0.5.4 Release vpatches. After sync'ing, we iterate through each vpatch and cat, and feeding them into `check_nonascii_bytes.pl`. Three bytes were discovered in the whole set of v0.5.4 Release vpatches, specifically 'e2', '80', and '94' from genesis.vpatch, in the source file of 'util.h'; the very same hyphen as aforementioned in this post.<br /><br />The following is source code for `check_nonascii_bytes.pl`:<br /><br /><pre><br />#!/usr/bin/perl<br /><br />use strict;<br /><br />my @vpatch=<STDIN>;<br /><br /># Keep track of the current source file we're looking through.<br />my $src_file = "";<br /><br /># Iterate over the vpatch one line at a time.<br />foreach my $line (@vpatch) {  <br /><br />  if($line =~ /^--- (.*) .*/) {<br />    $src_file = $1;<br />  }<br /><br />  # unpack the line into hex i.e. ;<br />  # 64696666202d754e7220612f626974636f696e2f2e67697469676e6f726520622f626974636f696e2f2e67697469676e6f72650a<br />  my $hex_line = unpack "H*", $line;<br /><br />  # add spaces between each two character hex byte, i.e. ;<br />  # 64 69 66 66 20 2d 75 4e 72 20 61 2f 62 69 74 63 6f 69 6e 2f 2e 67 69 74 69 67 6e 6f 72 65 20 62 2f 62 69 74 63 6f 69 6e 2f 2e 67 69 74 69 67 6e 6f 72 65 0a<br />  $hex_line =~ s/(..)(?!$)/$1 /g;<br /><br />  # Split out each two character hex representation by spaces, into array.<br />  my @line_bytes = split / /, $hex_line;<br />  <br />  # Iterate through the array of two character representations of bytes, one at a time.<br />  foreach my $lb (@line_bytes) {<br />    <br />    # Check the current two character representations.<br />    # First character must be between 0 and 7<br />    # Second character can be 0-9, or a-f.<br /><br />    # If the two character byte representation is *NOT* in this range, <br />    # we've found a non-seven-bit-ascii char.<br />    if($lb !~ /^[0-7][0-9a-f]$/i) { <br />      print "NON-7BIT-ASCII VALUE IN SOURCE FILE: $src_file ; BYTE VALUE: $lbn";<br />    }<br />  }<br />}<br /></pre><br /><br />Next, I wrote a C program that completes the same task, here's what it's execution looks like:<br /><br /><pre><br />mod6@gentoo ~/check_nonascii_bytes $ for i in `ls patches`; do echo "vpatch: $i"; ./cb patches/$i; done<br />vpatch: asciilifeform-kills-integer-retardation.vpatch<br />vpatch: asciilifeform_add_verifyall_option.vpatch<br />vpatch: asciilifeform_and_now_we_have_block_dumper_corrected.vpatch<br />vpatch: asciilifeform_and_now_we_have_eatblock.vpatch<br />vpatch: asciilifeform_dns_thermonyukyoolar_kleansing.vpatch<br />vpatch: asciilifeform_dnsseed_snipsnip.vpatch<br />vpatch: asciilifeform_lets_lose_testnet.vpatch<br />vpatch: asciilifeform_maxint_locks_corrected.vpatch<br />vpatch: asciilifeform_orphanage_thermonuke.vpatch<br />vpatch: asciilifeform_tx-orphanage_amputation.vpatch<br />vpatch: asciilifeform_ver_now_5_4_and_irc_is_gone_and_now_must_give_ip.vpatch<br />vpatch: asciilifeform_zap_hardcoded_seeds.vpatch<br />vpatch: asciilifeform_zap_showmyip_crud.vpatch<br />vpatch: bitcoin-asciilifeform.1.vpatch<br />vpatch: bitcoin-asciilifeform.2-https_snipsnip.vpatch<br />vpatch: bitcoin-asciilifeform.3-turdmeister-alert-snip.vpatch<br />vpatch: bitcoin-asciilifeform.4-goodbye-win32.vpatch<br />vpatch: bitcoin-v0_5_3-db_config.6.vpatch<br />vpatch: bitcoin-v0_5_3_1-rev_bump.7.vpatch<br />vpatch: bitcoin-v0_5_3_1-static_makefile_v002.8.vpatch<br />vpatch: genesis.vpatch<br />OUT OF 7-BIT ASCII RANGE! : e2<br />OUT OF 7-BIT ASCII RANGE! : 80<br />OUT OF 7-BIT ASCII RANGE! : 94<br />vpatch: makefiles.vpatch<br />vpatch: malleus_mikehearnificarum.vpatch<br />vpatch: mod6_der_high_low_s.vpatch<br />vpatch: mod6_fix_dumpblock_params.vpatch<br />vpatch: programmable-versionstring.vpatch<br />vpatch: rm_rf_upnp.vpatch<br /></pre><br /><br />Just as before, we iterate through each vpatch and feed it into our program, `cb` (check_bytes.c). Three bytes were discovered in the whole set of v0.5.4 Release vpatches, specifically 'e2', '80', and '94' from genesis.vpatch; the very same hyphen as aforementioned in this post.<br /><br />The following is source code for `check_bytes.c`:<br /><br /><pre><br />#include &lt;stdio.h&gt;<br />#include &lt;stdlib.h&gt;<br /><br />void read_file(char *filename) {<br /><br />  FILE      *file;<br />  char      *line  = NULL;<br />  size_t    len    = 0;<br />  ssize_t   read;<br /><br />  int pos          = 0;  // Position in line<br />  short int d      = 0;  // holds the short decimal value of the char<br /><br />  // attempt to open our file, read-only mode.<br />  file = fopen(filename, "r");<br /><br />  // if our file stream is NULL, print error and exit.<br />  if (file == NULL) {<br />    fprintf(stderr, "FILE COULD NOT BE OPENED! EXITING.n");<br />    exit(-1);<br />  }<br /><br />  // Read a file line by line.<br />  while((read = getline(&line, &len, file)) != -1) {<br /><br />    pos = 0;<br /><br />    // Iterate through each character in the line, until we hit the NULL byte.<br />    while(line[pos] != '\0') {<br /><br />      // set the current line position char in numeric form<br />      d = (short int) line[pos];<br /><br />      // test numeric form of char if within 7-BIT ASCII range<br />      if(d < 0x0 || d > 0x7f) {<br />        fprintf(stdout, "OUT OF 7-BIT ASCII RANGE! : %02xn", (unsigned char) line[pos]);<br />      }<br /><br />      // move to the next position in the line<br />      pos++;<br />    }<br />  }<br /><br />  // Clean up<br />  free(line);<br />  fclose(file);<br />}<br /><br />int main(int argc, char **argv) {<br />  int i;<br /><br />  // Iterate through all of the files given to us via command line<br />  for(i = 1; i < argc; i++) {<br />    read_file(argv[i]); // Call read_file for each given filename<br />  }<br /><br />  return 0;<br />}<br /></pre>