Difference between revisions of "Wiimms mkw-ana (tool)"

From Custom Mario Kart
Jump to navigation Jump to search
m (Wrong version)
(mkw-ana v0.07)
Line 1: Line 1:
 +
{| class="textbox float-right grid"
 +
|+ Wiimms mkw-ana
 +
| colspan=2 style="text-align:center" | [[file:Wiimms-SZS-Tools.png]]
 +
|-
 +
! Author:
 +
| [[user:Wiimm|Wiimm]]
 +
|-
 +
! Operating Systems:
 +
| Linux (i386,x86_64),<br/>Windows (cygwin).
 +
|-
 +
! Software Type:
 +
| Mario Kart Wii network analyzer
 +
|-
 +
! File Formats:
 +
| PCAP 2.4 (tcpdump), [[BMG]] (text).
 +
|-
 +
! Current Version:
 +
| 0.07, 2013-02-24
 +
|}
 +
 
The tools '''mkw-ana''' is a new project by [[Wiimm]] to analyze the network protocol of [[Mario Kart Wii]].
 
The tools '''mkw-ana''' is a new project by [[Wiimm]] to analyze the network protocol of [[Mario Kart Wii]].
  
Line 5: Line 25:
  
  
−
{{textbox|red center|In Progress|This article is managed by [[Wiimm]] will grow slowly. Feel free to edit typos and grammatical or speech issues.}}
+
{{textbox|red center|In Progress|This article is managed by [[Wiimm]], the developer, and will grow slowly. Feel free to edit typos and grammatical or speech issues.}}
−
 
 
  
 
== Intention ==
 
== Intention ==
Line 19: Line 38:
  
 
== The Tool ==
 
== The Tool ==
 +
 +
=== Change log ===
 +
 +
<spoiler><pre>
 +
mkw-ana v0.00 r1174 - 2012-12-06
 +
 +
- Start of the project.
 +
 +
 +
mkw-ana v0.01 r1177 - 2012-12-06
 +
 +
- Understanding a PCAP v2.4 dump and filtering relevant data.
 +
- Live logger of race data.
 +
 +
 +
mkw-ana v0.02 r1308 - 2013-01-05
 +
 +
- Command driven tool:
 +
    Commands: HELP, DEVELOP, TEST, DUMP1, DUMP2, DUMP3, DUMPX, DUMP, CLIENTS,
 +
      USERS, RECORDS, STAGES, LOG, RACE.
 +
    Options: --help --width --sleep --old --new --receive --send --length
 +
    --index --stage --type --TYPE --brief --long --one-line --ascii
 +
    --hex-only --log --md --mdx --mii dir     
 +
 +
mkw-ana v0.03 r1343 - 2013-01-11
 +
 +
- New command: FLOWRATES: Print flowrates of the data stream.
 +
- New command: TOTALS: Print only the final total tables of the LOG output.
 +
- New options: --skip=float --term?float --combine: Use only parts of the
 +
  input dumps and combine It logical to one stream.
 +
- New option: --write=file: Write the filtered dump to a new file.
 +
- New option: --home: Define an IP as home client. If not set, the first
 +
  sender is used. And this is sometimes wrong.
 +
- New option: --rel: Print relative time stamps.
 +
 +
 +
mkw-ana v0.04 r1370 - 2013-01-29
 +
- New record types: RACE_HEAD, RACE_DATA1, RACE_DATA2: Split the old race
 +
  data into a header and 1 or to player data records.
 +
- Optimized stage detecting.
 +
- New option: -p --no-proxy: Ignore proxy records.
 +
- New option: -f --follow: Follow last input file like 'tail -f'.
 +
- New options: --php --phpx: Create a php script with racing results after
 +
  each race.
 +
 +
 +
mkw-ana v0.05 r1414 - 2013-02-08
 +
- Different stages renamed.
 +
- Optimized stage detecting.
 +
- New record types: CONNECT, HELLO, ANNOUNCE: Used for handshaking.
 +
- New record types: STATUS_* (00|04|08|0C|10): Different status messages
 +
  about active players and slot arrangements.
 +
- New stages: SELECT_* (0|1|1S|2): Track selection before race.
 +
- Command LOG and md/php export: Show user name and index of voted and
 +
  selected tracks.
 +
- New option: -2 --sep-lines: Dump one line per record & an empty line
 +
  between records.
 +
 +
 +
mkw-ana v0.06 r1453 - 2013-02-20
 +
 +
- New rcord types: PREFIX_* (0..4): 4 different record variants before a
 +
  USER record: 0,3: unknown, 1:start an event, 3:send a room message.
 +
- New stages: START_EVENT: Start of an event (GP, TEAM, BATTLE or COIN).
 +
- New options: -q --quiet -v --verbose: Decrease and increase verbose level.
 +
- New option: DUMP3 -d --delta: Print unchanged bytes as '--'.
 +
- New option: --bmg=file: Read a BMG text file to scan online chat messages
 +
  and track, drive and vehicle names and disable auto load of BMG files.
 +
- New option: --team=file: Read a text file for team assignments and disable
 +
  auto load of team files.
 +
- Command LOG and md/php export: Show user and name of selected tracks.
 +
- Command LOG: Show driver and vehicle for each player (Miis are buggy).
 +
 +
 +
mkw-ana v0.07 r1475 - 2013-02-24
 +
 +
- New user interface with an extended and command dependent built-in help.
 +
  Command only allow options with impact to the command.
 +
  Try "mkw-ana HELP", "mkw-ana HELP command" or "mkw-ana HELP OPTIONS".
 +
- Command LOG and md/php export: Show driver and vehicle for each player,
 +
  if a BMG with names is loaded.
 +
 +
 +
</pre></spoiler>
  
 
=== Built-in Help ===
 
=== Built-in Help ===
  
 
Let's start with the built-in help as an overview about the tool:
 
Let's start with the built-in help as an overview about the tool:
−
<pre>
+
 
 +
<spoiler><pre>
 
Usage: mkw-ana [option]... [dumpfile]...
 
Usage: mkw-ana [option]... [dumpfile]...
  
 
Commands:
 
Commands:
  
−
  HELP                  Print this help and exit.
 
−
  DEVELOP              Special development tests.
 
−
  TEST                  Test options.
 
  
−
  D1|DUMP1 [file]...    Packet based hex dumper.
+
mkw-ana v0.07/x86_64 r1475 -- Dirk Clemens -- 2013-02-24
−
  D2|DUMP2 [file]...    First record base hex dumper.
+
--------------------------------------------------------
−
  D3|DUMP3 [file]...    New and improved variant of DUMP2 (record based).
+
 
−
  D|DUMP [file]...      Use the best/latest dumping mode (depends on options).
+
mkw-ana : Analyze network dumps (created by tcpdump) and print summaries.
  
−
  F|FLOWRATES [file]... Print flowrates (all TCP dumps, not only MKWii).
+
Syntax: mkw-ana [option]... command [option|parameter|file]...
−
                        Use --brief for final statistics only.
 
  
−
  C|CLIENTS [file]...  List all clients.
+
Commands:
−
  U|USERS [file]...    List all users.
 
−
  R|RECORDS [file]...  Print all record names.
 
−
  S|STAGES [file]...    Print all stages.
 
−
  L|LOG [file]...      Print all stages including tables.
 
−
    TRACKS [file]...    Print all track selections.
 
−
  T|TOTALS [file]...    Print all totals as text dump.
 
  
−
   RACE [file]...        Live dump of senders race statitics to the screen.
+
   VERSION        : Print program name and version and exit.
 +
  HELP      | H  : Print help and exit. If the first non option is a valid
 +
                  command name, then a help for the given command is printed.
 +
                  Command 'HELP OPTIONS' prints a list with all options.
 +
  ARGTEST       : This debug command accepts all kinds of parameters and
 +
                  prints one line for each parameter.
 +
  TEST          : Test options: All options are allowed, some are printed.
  
−
Options:
+
  DUMP1    | D1 : Packet based hex dumper.
 +
  DUMP2    | D2 : First record base hex dumper.
 +
  DUMP3    | D3 : New and improved variant of DUMP2 (record based).
 +
  DUMP      | D  : Use the best/latest dumping mode (depends on options). At
 +
                  the moment DUMP is an alias for DUMP3.
  
−
   -h --help          : Print this help and exit.
+
   FLOWRATES | F  : Print flowrates for all network dumps, not only for MKWii.
−
  -w --width          : Define terminal width.
 
−
  -O --old            : Enable old features and disable --new.
 
−
  -N --new            : Enable new features and disable --old.
 
  
−
    --sleep float    : Sleep seconds after race finished.
+
  CLIENTS  | C  : List all clients.
−
    --skip float    : Skip first seconds of each tcpdump.
+
  USERS    | U : List all users.
−
                        Negative values are relative to the end (not stdin).
+
  RECORDS  | R : Print all record names.
−
    --term float    : Terminate each dump at timestamp (seconds).
+
  STAGES    | S  : Print all stages.
−
                        Negative values are relative to the end (not stdin).
+
   LOG      | L  : Print all stages including tables.
−
    --combine        : Logical combine dumps to one for --skip and --term.
+
  TRACKS        : Print all track selections.
−
   -W --write file    : Write filtered packets as PCAP to the 'file'.
+
  TOTALS    | T : Print all totals as text dump.
−
    --home addr      : Define an IP (or name) as home client.
 
−
    --wii addr      : Define an IP (or name) of the Wii. Only packets from
 
−
                        and to this address are analyzed. If --home is not set,
 
−
                        'addr' is also used to define the home client.
 
  
 +
  RACE          : Live dump of senders race statitics to the screen.
  
−
  -p --no-proxy      : Ignore proxy packets.
+
Type 'mkw-ana HELP command' to get command specific help.
−
  -r --receive        : Select received packets.
 
−
  -s --send          : Select send packets.
 
−
  -L --length ranges  : Select UDP packets by UDP data length.
 
−
  -S --stage list    : Select stage types, print only matched records.
 
−
  -t --type list      : Select record types, print packet if one record match.
 
−
  -T --TYPE list      : Select record types, print only matched records.
 
−
  -I --index ranges  : Dump only bytes selected by range list.
 
  
−
  -b --brief          : Suppress header info.
+
Global options:
−
  -l --long          : Extended header info.
 
−
    --rel            : Print timestamps as relative seconds.
 
−
  -1 --one-line      : Dump one line per record.
 
−
                        If set twice, bind race records to one record.
 
−
  -2 --sep-lines      : Dump one line per record & an empty line between records.
 
−
  -a --ascii          : Append ASCII character dump.
 
−
  -x --hex-only      : Suppress AI and dump only hex values.
 
−
  -f --follow        : Follow last input file like 'tail -f'
 
  
−
     --bmg  file     : Read a BMG text file to scan track names and disable
+
  -V --version     Stop parsing the command line, print a version info and
−
                        auto load of BMG files. Multiple usage allowed.
+
                    exit.
 +
  -h --help        Stop parsing the command line, print a help message and
 +
                    exit.
 +
     --xhelp        Stop parsing the command line and print a help message with
 +
                    all commands included. Exit after printing.
 +
    --width width  Define the width (number of columns) for help and some
 +
                    other messages and disable the automatic detection of the
 +
                    terminal width.
 +
  -q --quiet        Be quiet and print only error messages. All previous
 +
                    --verbose are canceled. Multiple usage is possible. The
 +
                    impact is command dependent.
 +
  -v --verbose      Be verbose and print more progress information. All
 +
                    previous --quiet are canceled. Multiple usage is possible.
 +
                    The impact is command dependent.
 +
  -A --allow-all    Usually commands accept only options with impact to the
 +
                    command. All other options firce an syntax error. But if
 +
                    --allow-all is set, all commands accept all options.
 +
    --old          Use old implementation if available. All previous --new are
 +
                    canceled.
 +
    --new          Use new implementation if available. All previous --old are
 +
                    canceled.
  
−
     --log file      : Log into the file. If first char is a '+', append data.
+
     --bmg file    Read a BMG text file to scan online chat messages and
−
     --md   file     : Create a MakeDoc script with results after each race.
+
                    track, drive and vehicle names. Disable auto load of BMG
−
     --mdx file      : Same as --md, but replace %E, %R, %N and %T in 'file'
+
                    files. Multiple usage allowed
−
                        by 'event id', 'race id', 'total race' and 'event type'.
+
    --team file    Read a text file for team assignments and disable auto load
−
     --php file      : Create a php script with results after each race.
+
                    of team files. Multiple usage allowed
−
     --phpx file     : Same as --php, but replace %E, %R, %N and %T in 'file'
+
 
−
                        by 'event id', 'race id', 'total race' and 'event type'.
+
Command specific options with common description:
−
     --mii dir       : Command USERS: Extract Miis to existing directory 'dir'.
+
 
−
</pre>
+
    --adjust float Adjust time stamps of the network dump by adding 'float'
 +
                    seconds. This may help to synchronize different dumps.
 +
    --skip float    Skip first 'float' seconds of each read network dump.
 +
                    Negative values are relative to the end (or ignored for
 +
                    pipes).
 +
    --term float    Terminate each dump at 'float' seconds. Negative values
 +
                    are relative to the end (or ignored for pipes).
 +
    --combine      Logical combine network dumps to one single dump before
 +
                    executing options --skip and --term.
 +
    --follow        Don't close the last input dump on reaching end of file.
 +
                    Instead wait for appended data. This works like the unix
 +
                    tool 'tail -f'.
 +
    --ip addr      Define an IP (or DNS name) for filtering. Only packets
 +
                    from an to this host are accepted, all others are ignored.
 +
    --home addr    Define an IP (or DNS name) as home client.
 +
                      Without this options, the tool tries to determine the
 +
                    home client by analysing sender and receiver of the first
 +
                    non filtered packet. A local network (10/8, 172.16/12,
 +
                    192.168/16, 169.254/16) has priority over a non local
 +
                    network. If sender and receiver have the same priority,
 +
                    the sender is used.
 +
    --wii addr      Define an IP (or DNS name) ad home client and for
 +
                    filtering. This options is a shortcut for '--ip addr
 +
                    --home addr'.
 +
    --write file    Write filtered network packets as PCAP to the 'file'.
 +
 
 +
  -p --no-proxy     Ignore proxy packets.
 +
  -r --receive      Select only packtes received by the home client. If non or
 +
                    both of --receive and --send are set, this filter is
 +
                    deactivated.
 +
  -s --send          Select only packtes send by the home client. If non or
 +
                    both of --receive and --send are set, this filter is
 +
                    deactivated.
 +
  -L --length ranges Select UDP packets by UDP data length. The 8 bytes long
 +
                    UDP header does not count.
 +
                      The parameter is a comma separated list of INDEX,
 +
                    INDEX1:, INDEX1:INDEX2 and INDEX#LENGTH elements.
 +
  -S --stage list    Accept UDP packets only, if one of the entered stages is
 +
                    active.
 +
                      The parameter is a comma separated list of stage names.
 +
                    A stage list is not available yet; use the dumps to
 +
                    identify stage names.
 +
  -t --type list    Select UDP packets by record types. The complete packet is
 +
                    selected if at least one of the entered match.
 +
                      The parameter is a comma separated list of record names.
 +
                    A record list is not available yet; use the dumps to
 +
                    identify record names.
 +
  -T --TYPE list    Same as --type except for command DUMP3.
 +
 
 +
  -b --brief        If set once, the dump header (timestamp and client info)
 +
                    of single line dumps becomes smaller. If set twice,
 +
                    timestamp and client info are not printed. All previous
 +
                    --long are canceled.
 +
  -l --long          This option is relevant for single line dumps. usally the
 +
                    time format is printed as 'MM:SS.s' to keep the lines
 +
                    small. If set once, 'HH:MM:SS.s' is used. If set twice,
 +
                    'HH:MM:SS.sss' is used. All previous --brief are canceled.
 +
    --rel          Print timestamps as seconds relative to the beginning.
 +
                    Dependent of option --long the foramts are: 'SSSSS',
 +
                    'SSSSS.s' or 'SSSSS.sss'
 +
  -1 --one-line      Print the hexdumps as one line for each object. This makes
 +
                    the dumps horizontal very large, but is good for comparing
 +
                    objects of the same type. Very helpful is to pipe the
 +
                    output to 'less -S', which supports horizontal scrolling.
 +
                      If set twice, some record types are not split into sub
 +
                    records.
 +
  -2 --sep-lines    Dump one line per record (like option --one-line) an empty
 +
                    line between packtes.
 +
  -a --ascii        Append an ASCII character dump behind the hexdump. This
 +
                    option is ignored if using --one-line.
 +
  -x --hex-only      Some known values a printed in native format instead as
 +
                    hex number. If option --hex-only is set, all data is
 +
                    printed as hex number.
 +
  -d --delta        If set, record data is comapred with the data of the
 +
                    previous record of same type and client. If a byte is
 +
                    unchanged, a '--' is printed inted of 2 hex digits.
 +
  -I --index ranges  Dump only bytes selected by the range list. This make the
 +
                    hex dump smaller especially for one-line dumps.
 +
                      The parameter is a comma separated list of INDEX,
 +
                    INDEX1:, INDEX1:INDEX2 and INDEX#LENGTH elements.
 +
 
 +
    --log file      Log into the file using the same output as command LOG. If
 +
                    first character of 'file' is a '+', append data to an
 +
                    already existend file.
 +
     --md file       Create a MakeDoc script with results after each race.
 +
     --mdx file      Create a MakeDoc script with results after each race. Same
 +
                    as --md, but replace %E, %R, %N and %T in the filename by
 +
                    'event id', 'race id', 'total race' and 'event type' to
 +
                    create different files.
 +
     --php file      Create a php script with results after each race.
 +
     --phpx file     Create a php script with results after each race. Same as
 +
                    --php, but replace %E, %R, %N and %T in the filename by
 +
                    'event id', 'race id', 'total race' and 'event type' to
 +
                    create different files.
 +
    --sleep float  Sleep 'float' seconds after a race has finished. This
 +
                    options slow down a simulation direct after logging and
 +
                    printing the race statistics.
 +
 
 +
     --mii dir       Extract Miis to the existing directory 'dir'. Existing Mii
 +
                    files will be overwritten.
 +
</pre></spoiler>
  
 
=== General Description ===
 
=== General Description ===
Line 131: Line 334:
 
=== Options ===
 
=== Options ===
 
=== Download ===
 
=== Download ===
−
Here is the v0.05 of the tool mkw-ana:
+
Here is the v0.07 of the tool mkw-ana:
−
[http://download.wiimm.de/mkw-ana/mkw-ana-v0.05-r1429.zip mkw-ana-v0.05-r1429.zip] (240 kB)
+
[http://download.wiimm.de/mkw-ana/mkw-ana-v0.07-r1475.zip mkw-ana-v0.07-r1475.zip] (305 kB)
 
* It contains binaries for:
 
* It contains binaries for:
 
** Linux i386
 
** Linux i386
Line 138: Line 341:
 
** Cygwin/Windows (A Cygwin system must be installed).
 
** Cygwin/Windows (A Cygwin system must be installed).
 
* Some scripts as examples.
 
* Some scripts as examples.
 +
* Some BMG text examples.
 +
* Some doc files.
  
 
'''It is an alpha distribution and and only minimal support will be given at the moment.'''
 
'''It is an alpha distribution and and only minimal support will be given at the moment.'''

Revision as of 18:07, 24 February 2013

Wiimms mkw-ana
File:Wiimms-SZS-Tools.png
Author: Wiimm
Operating Systems: Linux (i386,x86_64),
Windows (cygwin).
Software Type: Mario Kart Wii network analyzer
File Formats: PCAP 2.4 (tcpdump), BMG (text).
Current Version: 0.07, 2013-02-24

The tools mkw-ana is a new project by Wiimm to analyze the network protocol of Mario Kart Wii.



In Progress
This article is managed by Wiimm, the developer, and will grow slowly. Feel free to edit typos and grammatical or speech issues.

Intention

In November 2012, Wiimm decided to analyze the network protocol of Mario Kart Wii. The main goal was to discover online cheaters. Another goal is to set up his own server if any time Nintendo will shut down its servers.

So the main feature is to dump the packets of a tcpdump (done by tcpdump or wireshark) in a user-friendly format. After first experiments, it becomes also a live racing statistic tool.

Terms and definition

Template:MKWii Network Terms

The Tool

Change log

Show text


Built-in Help

Let's start with the built-in help as an overview about the tool:

Show text


General Description

The tool started as simple hex dumper reading network dumps in PCAP format. In the first phase of the tool, the textual dumps of whireshark and tcpdump were much better. But after only a few days, the tool learned to handle records, clients, users, friend codes and Miis. From this moment the tool was better to analyze the Mario Kart Wii traffic.

Now, mkw-ana split the traffic into records and scans some data to detect stages of the online meeting. Stages are for example room, prepare race, count down, racing and end of race. It is able to separate races into events (grand prix and team rand prix) and to calculate racing tables. Racing data can also be exported to support live statistics.

At the moment there are three different kinds of hexdumps. All 3 are able to dump in one line mode to have large tables. Tool less is here a very good tool for vertical and horizontal scrolling. The stages are includes into the dump as comment lines. The dumped records can be filters by sending, receiving, proxy, record types, stage types and packet length. It is also possible to select the dumped bytes by indices and ranges.

Another feature is, that mkw-ana can read comment files. If making videos of the dumped meetings, you can write such comment file. Each line starts with a timestamp followed by a comment. Virtual Dub is a good tool for this job. Then you must synchronize the comment file with the network dump. The start of the first game ("GO" in the video) is a very good point for synchronization. Here is an example of a comment file (in german): 

>2012-12-05 19:10:39.745 - 12:22.792
0:00:00.000 Video Start

0:12:22.792 Rennen 1.1, GO!
0:12:54.123 Tinti wird angekündigt, T=0:31.322
0:14:17.924 Power wird angekündigt, T=1:55.120
0:15:33.633 Blitz schlägt ein, T=1:35.326
...
Notes
  • The first line ist the real time of the start of the race minus the video time stamp. This is the synchronisation. An synchronisation can be done multiple times.
  • The line with video timestamp and comment follow. The focus of the comment change as the point of interest.

???

Dumping Data

Options

Download

Here is the v0.07 of the tool mkw-ana: mkw-ana-v0.07-r1475.zip (305 kB)

  • It contains binaries for:
    • Linux i386
    • Linux x86_64
    • Cygwin/Windows (A Cygwin system must be installed).
  • Some scripts as examples.
  • Some BMG text examples.
  • Some doc files.

It is an alpha distribution and and only minimal support will be given at the moment.

Capture the network data

First you must capture the network traffic of the Wii. Therefore you must redirect it to a PC running a capture software. There are 3 general ways to to this:

  • If you have a manageable switch, enable port mirroring and send all Wii traffic to a PC.
  • Use your PC as router.
  • Use old network hubs (not switches). A hub will mirror all traffic of all ports to all others; it's just a multi port repeater and will slow down your network.

Use a software like tcpdump or wireshark to capture the data. Best is to save the captured data directly to a file or to send it to other commands like mkw-ana for a live analysis.

It's also possible to save the data to a file and to make a live analysis at the same time. Use the following command pipe: 

tcpdump -w- -U -i eth1 host wii | tee save.dump | mkw-ana ...

If using wireshark, save the dump to a file and use the following command for a live analysis: 

mkw-ana --follow DUMPFILE ...

Live Statistics

mkw-ana can scan the network traffic in real time and can produce makedoc or php data files. Together with ssh and an cgi script, a live statistic is created. Live means that the tables are updates 2-5 seconds after the race have finished.

How it works

The whole job is done by 3 processes:

  1. First, you must capture the network traffic like described above. Then use one of the commands: 
    ... | mkw-ana log --md  DATAFILE
... | mkw-ana log --php DATAFILE
mkw-ana --follow DUMPFILE log --md  DATAFILE
mkw-ana --follow DUMPFILE log --php DATAFILE
  2. Each time, a new DATAFILE is written, it must be transferred to the web server. A script using shh, sftp scp or ftp within an endless loop will do this job automatically.
  3. Last not least, a CGI or PHP script running at the web server must read the data files to serve a html-page to the visitors.

To see, what live means, visit the live statistics on Wednesday or Thursday between 19:10 and 20:30 CET (Central European Time).

Links

Retrieved from "https://wiki.tockdom.com/w/index.php?title=Wiimms_mkw-ana_(tool)&oldid=32428"