Difference between revisions of "Wiimms mkw-ana (tool)"

From Custom Mario Kart
Jump to navigation Jump to search
(mkw-ana v0.16 released)
Line 16: Line 16:
 
|-
 
|-
 
! Current Version:
 
! Current Version:
−
| v0.16.beta4, 2014-01-12
+
| v0.16, 2014-01-18
 
|}
 
|}
  
Line 40: Line 40:
  
 
<pre>
 
<pre>
−
mkw-ana v0.16.beta4 r1737 - 2014-01-12
+
mkw-ana v0.16 r1765 - 2014-01-18
  
−
  - The tool accept now the following dump formats:
+
  - The tool accepts now the following dump formats:
−
     - PCAP v2.4: big and little endian, micro- and nanoseconds format,
+
     - PCAP v2.4: Big or little endian, micro- or nanoseconds format.
−
      standard or 'modified' format with 8 extra bytes.
+
    - PCAP v2.4 modified: Like PCAP, but with an extended packet header.
−
     - PCAP-NG v1.0 (experimental): both endians, enhanced packet block.
+
     - PCAP-NG v1.0: Big or little endian. Only the `Enhanced Packet Block´ is
−
     - Compression bzip2.
+
      supported to retrieve packets.  
 +
     - Optional bzip2 compression of all dump formats.
  
−
  - New global option: --wide[=width]: Usually hexdumps covers 16 bytes per
+
- Complete new handling of ARP, DNS and TCP packets. They will now be handled
 +
  as records and are written to the output file, if option --write is set.
 +
 
 +
- New record types: ARP, DNS, TCP, QUERY, TPARAM, UPARAM.
 +
 
 +
  - New global option: --wide[=width]: Usually hexdumps cover 16 bytes per
 
   line. If option --wide is set, 32 bytes per line are covered. Optional it
 
   line. If option --wide is set, 32 bytes per line are covered. Optional it
 
   is possible to enter a value.
 
   is possible to enter a value.
  
 
  - New keywords for option --log-mode:
 
  - New keywords for option --log-mode:
−
     'TCP'   : TCP packets are logged as hexdump (excluded from default).
+
     'QUERY' : Dump database quers (DB,table,select,where).
 
     'REGION' : Dump region and world wide queries.
 
     'REGION' : Dump region and world wide queries.
 +
    'TCP'    : Shortcut for QUERY,REGION: Log all TCP related stuff.
  
 
  - New options: Option --color forces colorized text (where supported).
 
  - New options: Option --color forces colorized text (where supported).
Line 62: Line 69:
 
  - New command: COLORS: Test colorized text by printing it in different modes
 
  - New command: COLORS: Test colorized text by printing it in different modes
 
   (colors, bold, underline). Also test the options --color and --no-color.
 
   (colors, bold, underline). Also test the options --color and --no-color.
 +
 +
- New command LIST: List all stage or record names.
 +
 +
- More options to filter packets for dumps: --and, --receive-mac (--rmac),
 +
  --send-mac (--smac) and --transfer-mac (--tmac).
 +
 +
- New options to control MAC printing in dumps: --show-mac and --hide-mac.
 +
 +
- The new options --real-time-factor (--rtf) and --real-time-wait (--rtw)
 +
  help to analyze old dumps in real time, time-laps or slow-motion.
  
  
Line 105: Line 122:
 
     XDRIFT=DRIFT+D-MINTIME, ITEM=I-CHEAT, CHEAT=D-CHEAT,I-CHEAT TEST, FC,
 
     XDRIFT=DRIFT+D-MINTIME, ITEM=I-CHEAT, CHEAT=D-CHEAT,I-CHEAT TEST, FC,
 
     MINI-FC and NAME. Also available: CLEAR, MIN, DEFAULT, MAX and ALL.
 
     MINI-FC and NAME. Also available: CLEAR, MIN, DEFAULT, MAX and ALL.
−
 
−
 
 
</pre></spoiler>
 
</pre></spoiler>
  
Line 114: Line 129:
  
 
<spoiler><pre>
 
<spoiler><pre>
−
 
+
mkw-ana v0.16/x86_64 r1765 -- Dirk Clemens -- 2014-01-18
−
mkw-ana v0.16.beta4/x86_64 r1737 -- Dirk Clemens -- 2014-01-12
+
--------------------------------------------------------
−
--------------------------------------------------------------
 
  
 
mkw-ana : Analyze network dumps (created by tcpdump) and print summaries.
 
mkw-ana : Analyze network dumps (created by tcpdump) and print summaries.
Line 133: Line 147:
 
   ERROR    | ERR : Translate exit codes to message names. If no exit code is
 
   ERROR    | ERR : Translate exit codes to message names. If no exit code is
 
                     entered, print a table with all error messages.
 
                     entered, print a table with all error messages.
 +
  LIST            : List the keywords of a class. Allowed classes are: STAGES,
 +
                    RECORDS and ALL.
  
 
   DUMP0    | D0  : Print a raw dump of all packets. This dump can be used for
 
   DUMP0    | D0  : Print a raw dump of all packets. This dump can be used for
Line 269: Line 285:
 
     --write file    Write filtered network packets as PCAP v2.4 to 'file' with
 
     --write file    Write filtered network packets as PCAP v2.4 to 'file' with
 
                     local endian and microseconds format.
 
                     local endian and microseconds format.
 +
    --real-time-factor factor
 +
                    If set (>0.0), the time differences of the packet time is
 +
                    compared with the real time difference. If a packet will
 +
                    be served to early, the tool sleeps a while.
 +
                      Value 1.0 force a real time dump. Values >1.0 force a
 +
                    time-laps effect and values <1.0 a slow-motion effect.
 +
                    --rtf is a short cut.
 +
                      The intention of this option is to simulate a regular
 +
                    input stream on already dumped and stored data in real
 +
                    time. Use this option never for live incoming data,
 +
                    because packets may be lost.
 +
    --real-time-wait seconds
 +
                    If set (>0.0) and the real time option --real-time-factor
 +
                    is enabled, it defines the maximum real time between 2
 +
                    packets. The default is 5 seconds. --rtw is a short cut.
  
 
   -p --no-proxy      Don't dump proxy packets (packets, which contains a PROXY
 
   -p --no-proxy      Don't dump proxy packets (packets, which contains a PROXY
 
                     record).
 
                     record).
 +
    --and          If one or more filters are enabled by --receive, --send,
 +
                    --receive-mac, --send-mac, --receive-ip or --send-ip, then
 +
                    a packet or record is only dumped, if it match to at least
 +
                    one of the enabled filters.
 +
                      But if --and is set, a packet must match *all* enabled
 +
                    filters.
 
   -r --receive      Dump only network packets received by the home client
 
   -r --receive      Dump only network packets received by the home client
−
                     (option --home) or allowed by --send, --receive-ip or
+
                     (option --home). For combinations with other packet
−
                     --send-ip.
+
                     filters see option --and.
 
   -s --send          Dump only network packets send by the home client (option
 
   -s --send          Dump only network packets send by the home client (option
−
                     --home) or allowed by --receive, --send-ip or
+
                     --home). For combinations with other packet filters see
−
                     --receive-ip.
+
                    option --and.
 +
    --receive-mac addr
 +
                    Dump only network packets received by the entered MAC
 +
                    address. --rmac is a short cut for --receive-mac. For
 +
                    combinations with other packet filters see option --and.
 +
    --send-mac addr Dump only network packets send by the entered MAC address.
 +
                    --smac is a short cut for --send-mac. For combinations
 +
                    with other packet filters see option --and.
 +
    --transfer-mac addr
 +
                    Dump only network packets receiced or send by the entered
 +
                     MAC address. --tmac is a short cut for --transfer-mac and
 +
                    both are short cuts for '--rmac addr --smac addr'.
 
     --receive-ip addr  
 
     --receive-ip addr  
 
                     Dump only network packets received by the entered address
 
                     Dump only network packets received by the entered address
−
                     (IP or DNS name) or allowed by --send-ip, --receive or
+
                     (IP or DNS name). --rip is a short cut for --receive-ip.
−
                    --send. --rip is a short cut for --receive-ip.
+
                    For combinations with other packet filters see option
 +
                    --and.
 
     --send-ip addr  Dump only network packets send by the entered address (IP
 
     --send-ip addr  Dump only network packets send by the entered address (IP
−
                     or DNS name) or allowed by --receive-ip, --send or
+
                     or DNS name). --sip is a short cut for --send-ip. For
−
                    --receive. --sip is a short cut for --send-ip.
+
                    combinations with other packet filters see option --and.
 
     --transfer-ip addr  
 
     --transfer-ip addr  
 
                     Dump only network packets receiced or send by the entered
 
                     Dump only network packets receiced or send by the entered
−
                     address (IP or DNS name) or allowed by --send or
+
                     address. --tip is a short cut for --transfer-ip and both
−
                    --receive. --tip is a short cut for --transfer-ip and both
+
                     are short cuts for '--rip addr --sip addr'.
−
                     are short cuts for '--rip addr --sip addr.
 
 
   -L --length ranges Dump only UDP packets with specified UDP data length. The
 
   -L --length ranges Dump only UDP packets with specified UDP data length. The
 
                     8 bytes long UDP header does not count.
 
                     8 bytes long UDP header does not count.
Line 312: Line 360:
 
   -T --TYPE list    Same as --type except for command DUMP3.
 
   -T --TYPE list    Same as --type except for command DUMP3.
  
−
   -b --brief        If set once, the dump header (timestamp and client info)
+
   -b --brief        If set once, the header (timestamp and client info) of
−
                     of single line dumps becomes smaller. If set twice,
+
                     single line dumps becomes smaller. If set twice, timestamp
−
                     timestamp and client info are not printed. All previous
+
                     and client info are not printed. All previous --long are
−
                    --long are canceled.
+
                    canceled.
 
   -l --long          This option is relevant for single line dumps. Usually the
 
   -l --long          This option is relevant for single line dumps. Usually the
 
                     time format is printed as 'MM:SS.s' to keep the lines
 
                     time format is printed as 'MM:SS.s' to keep the lines
Line 330: Line 378:
 
   -2 --sep-lines    Dump one line per record (like option --one-line) and an
 
   -2 --sep-lines    Dump one line per record (like option --one-line) and an
 
                     empty line between packets.
 
                     empty line between packets.
 +
    --show-mac      Show the MAC addresses of packets in some dumps. This is
 +
                    enabled by default, if at least one MAC packet filter
 +
                    (--receive-mac or {--send-mac) is enabled.
 +
    --hide-mac      Hide the MAC addresses of packets in all dumps. This is
 +
                    the default, if no MAC packet filter is enabled.
 
   -n --native        If set, some known values are printed in native format
 
   -n --native        If set, some known values are printed in native format
 
                     instead as simple hex number. If set twice, some other
 
                     instead as simple hex number. If set twice, some other
 
                     values, that will destroy the column layout of the
 
                     values, that will destroy the column layout of the
 
                     hexdump, will printed in native format too.
 
                     hexdump, will printed in native format too.
 +
  -x --hex          Some records are printed as hex and string combination by
 +
                    default. If --hex is set, then print the3se records as hex
 +
                    dumps.
 
   -d --delta        If set, record data is compared with the data of the
 
   -d --delta        If set, record data is compared with the data of the
 
                     previous record of same type and client. If a nibble (4
 
                     previous record of same type and client. If a nibble (4
Line 353: Line 409:
 
     --min-race num  This is a statistic option: If a Grand Prix (single or
 
     --min-race num  This is a statistic option: If a Grand Prix (single or
 
                     team) is aborted, the results of the Grand Prix are only
 
                     team) is aborted, the results of the Grand Prix are only
−
                     used in the statstics, if NUM races has been completed.
+
                     used in the statistics, if NUM races has been completed.
 
                     The default is 2 and possible values are 0..4.
 
                     The default is 2 and possible values are 0..4.
 
     --drift        Print drift statistics during logging.
 
     --drift        Print drift statistics during logging.
Line 364: Line 420:
 
                     --log). --lmd is a short cut. A comma separated list of
 
                     --log). --lmd is a short cut. A comma separated list of
 
                     keywords is expected: STATUS, SELECT, DRIVER, RACE, EVENT,
 
                     keywords is expected: STATUS, SELECT, DRIVER, RACE, EVENT,
−
                     TOTAL, CHEATS, TCP, REGION, DEFAULT, NONE and ALL.
+
                     TOTAL, CHEATS, TCP, QUERY, REGION, DEFAULT, XTCP, NONE and
 +
                    ALL.
 
     --md file      Create a MakeDoc script with results after each race.
 
     --md file      Create a MakeDoc script with results after each race.
 
     --mdx file      Create a MakeDoc script with results after each race. Same
 
     --mdx file      Create a MakeDoc script with results after each race. Same
Line 382: Line 439:
 
     --mii dir      Extract Miis to the already existing directory 'dir'.
 
     --mii dir      Extract Miis to the already existing directory 'dir'.
 
                     Existing Mii files will be overwritten.
 
                     Existing Mii files will be overwritten.
 +
 
</pre></spoiler>
 
</pre></spoiler>
  

Revision as of 08:31, 18 January 2014

Wiimms mkw-ana
File:Wiimms-SZS-Tools.png
Author: Wiimm
Operating Systems: Linux (i386,x86_64),
Windows (Cygwin).
Software Type: Mario Kart Wii network traffic analyzer
File Formats: PCAP 2.4, PCAP 2.4.modified, PCAP-NG, bzip2 compression, BMG (text).
Current Version: v0.16, 2014-01-18

The tools mkw-ana is a new project by Wiimm to analyze the network protocol of Mario Kart Wii.



Intention

In November 2012, Wiimm decided to analyze the network protocol of Mario Kart Wii. The main goal was to discover online cheaters. Another goal is to set up his own server if any time Nintendo will shut down its servers.

So the main feature is to dump the packets of a tcpdump (done by tcpdump or wireshark) in a user-friendly format. After first experiments, it becomes also a live racing statistic tool.

Terms and definition

Template:MKWii Network Terms

The Tool

Change log

mkw-ana v0.16 r1765 - 2014-01-18

 - The tool accepts now the following dump formats:
    - PCAP v2.4: Big or little endian, micro- or nanoseconds format.
    - PCAP v2.4 modified: Like PCAP, but with an extended packet header.
    - PCAP-NG v1.0: Big or little endian. Only the `Enhanced Packet Block´ is
      supported to retrieve packets. 
    - Optional bzip2 compression of all dump formats.

 - Complete new handling of ARP, DNS and TCP packets. They will now be handled
   as records and are written to the output file, if option --write is set.

 - New record types: ARP, DNS, TCP, QUERY, TPARAM, UPARAM.

 - New global option: --wide[=width]: Usually hexdumps cover 16 bytes per
   line. If option --wide is set, 32 bytes per line are covered. Optional it
   is possible to enter a value.

 - New keywords for option --log-mode:
     'QUERY'  : Dump database quers (DB,table,select,where).
     'REGION' : Dump region and world wide queries.
     'TCP'    : Shortcut for QUERY,REGION: Log all TCP related stuff.

 - New options: Option --color forces colorized text (where supported).
   It is enabled by default for output to terminals. Option --no-color
   disables colorized text at all.

 - New command: COLORS: Test colorized text by printing it in different modes
   (colors, bold, underline). Also test the options --color and --no-color.

 - New command LIST: List all stage or record names.

 - More options to filter packets for dumps: --and, --receive-mac (--rmac),
   --send-mac (--smac) and --transfer-mac (--tmac).

 - New options to control MAC printing in dumps: --show-mac and --hide-mac.

 - The new options --real-time-factor (--rtf) and --real-time-wait (--rtw)
   help to analyze old dumps in real time, time-laps or slow-motion.


mkw-ana v0.15 r1703 - 2014-01-04

 - Cup index is now read from BMG.
 - Update of BMG files.
 - New option --log-mode=list (or short --lmd=list): Define, which elements
   are included into the log file or output. Allowed keywords are: STATUS,
   SELECT, DRIVER, RACE, EVENT, TOTAL, CHEATS, NONE and ALL (default).
Old change log

Show text


Built-in Help

Let's start with the built-in help as an overview about the tool:

Show text


General Description

The tool started as simple hex dumper reading network dumps in PCAP format. In the first phase of the tool, the textual dumps of wireshark and tcpdump were much better. But after only a few days, the tool learned to handle records, clients, users, friend codes and Miis. From this moment the tool was better to analyze the Mario Kart Wii traffic.

Now, mkw-ana split the traffic into records and scans some data to detect stages of the online meeting. Stages are for example room, prepare race, count down, racing and end of race. It is able to separate races into events (grand prix and team rand prix) and to calculate racing tables. Racing data can also be exported to support live statistics.

At the moment there are three different kinds of hexdumps. All 3 are able to dump in one line mode to have large tables. Tool less is here a very good tool for vertical and horizontal scrolling. The stages are includes into the dump as comment lines. The dumped records can be filters by sending, receiving, proxy, record types, stage types and packet length. It is also possible to select the dumped bytes by indices and ranges.

Another feature is, that mkw-ana can read comment files. If making videos of the dumped meetings, you can write such comment file. Each line starts with a timestamp followed by a comment. Virtual Dub is a good tool for this job. Then you must synchronize the comment file with the network dump. The start of the first game ("GO" in the video) is a very good point for synchronization. Here is an example of a comment file (in german): 

>2012-12-05 19:10:39.745 - 12:22.792
0:00:00.000 Video Start

0:12:22.792 Rennen 1.1, GO!
0:12:54.123 Tinti wird angekündigt, T=0:31.322
0:14:17.924 Power wird angekündigt, T=1:55.120
0:15:33.633 Blitz schlägt ein, T=1:35.326
...
Notes
  • The first line is the real time of the start of the race minus the video time stamp. This is the synchronisation. An synchronisation can be done multiple times.
  • The line with video timestamp and comment follow. The focus of the comment change as the point of interest.
  • The name of the comment file must be the same as the network dump, but it must have the extension ".info" instead of ".eth".

???

Dumping Data

Options

Download

You can find the latest and some old distributions here
Content
  • Binaries for:
    • Linux i386
    • Linux x86_64
    • Cygwin/Windows (Needed Cygwin[1] DLL files are delivered. Best is to install a Cygwin system).
  • Some scripts as examples.
  • Some BMG text examples.
  • Some doc files.
Sometimes I upload single tool updates (beta versions) for testers

Capture the network data

First you must capture the network traffic of the Wii. Therefore you must redirect it to a PC running a capture software. There are 3 general ways to to this:

  • If you have a manageable switch, enable port mirroring and send all Wii traffic to a PC.
  • Use your PC as router.
  • Use old network hubs (not switches). A hub will mirror all traffic of all ports to all others; it's just a multi port repeater and will slow down your network.

Use a software like tcpdump or wireshark to capture the data. Best is to save the captured data directly to a file or to send it to other commands like mkw-ana for a live analysis.

It's also possible to save the data to a file and to make a live analysis at the same time. Use the following command pipe: 

tcpdump -w- -U -i eth1 host wii | tee save.dump | mkw-ana ...

It is important to filter the data by host ip_or_name, because foreign traffic interfere the wii traffic analysis and will have curious side effects.

If using wireshark, save the dump to a file and use the following command for a live analysis: 

mkw-ana --follow DUMPFILE ...

Accepted file formats

mkw-ana accepts the following file formats for the network dumps:

  • PCAP 2.4 : Standard packet capturing file format[2].
    • Big and little endian are supported.
    • Timestamps in micro- and in nanoseconds are supported.
  • PCAP 2.4.modified : Like PCAP, but with an extend packet header. This format is used by several routers, AVM FRITZ!Box[3] is one example.
    • Big and little endian are supported.
    • Only microseconds timestamps are supported.
  • PCAP-NG 2.4.modified : A next generation (NG) PCAP format[4].
    • Big and little endian are supported.
    • Microseconds timestamps are assumed, other are not supported.
    • Only the Enhanced Packet Block[5] is supported to retrieve packets.
  • BZIP2 compression
    • mkw-ana detects a BZIP2 compression automatically. It is supported for all other dump file formats.

mkw-ana accpets any list of dump files. The file format is detected for each single input file, so mixed formats are possible. The special file name »-« (minus sign) means: Don't open the file and and read the standard input (stdin) instead. So one of the input files can be read via pipe.

Live Statistics

mkw-ana can scan the network traffic in real time and can produce makedoc or php data files. Together with ssh and an cgi script, a live statistic is created. Live means that the tables are updates 2-5 seconds after the race have finished.

How it works

The whole job is done by 3 processes:

  1. First, you must capture the network traffic like described above. Then use one of the commands: 
    ... | mkw-ana log --md  DATAFILE
... | mkw-ana log --php DATAFILE
mkw-ana --follow DUMPFILE log --md  DATAFILE
mkw-ana --follow DUMPFILE log --php DATAFILE
  2. Each time, a new DATAFILE is written, it must be transferred to the web server. A script using ssh, sftp scp or ftp within an endless loop will do this job automatically.
  3. Last not least, a CGI or PHP script running at the web server must read the data files to serve a html-page to the visitors.

To see, what live means, visit the live statistics on Wednesday or Thursday between 19:10 and 20:30 CET (Central European Time).

Links

References

  1. ↑ Cygwin, a Linux like environment for Windows.
  2. ↑ Wireshark: LIBPCAP file format
  3. ↑ AVM FRITZ!Box, a widely used router in Germany
  4. ↑ Wincap: The next generation PCAP file format
  5. ↑ Wincap: PCAP-NG, Enhanced Packet Block


Template:MKWii Network Protocol

Retrieved from "https://wiki.tockdom.com/w/index.php?title=Wiimms_mkw-ana_(tool)&oldid=57045"